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ABSTRACT 

This report describes the design of a test cf one 
aspect of critical thinking ability, the ability to correctly 
appraise observations-. Intended for * classroom use with senior high 
school students, the 50 item Test on Appraising Observations is based 
on a comprehensive set of principles modified from Robert Ennes' 
conception of good observation appraisal. The test evolved throug*h 
many versions using methodology developed from the construct validity 
theory that ability tests are valid to the extent that good thinking* 
leads to good test performance and that poor thinking leads to bad 
test performance. The methodology involved the systematic collection 
of thinking about protocols of examinees while they worked through 
test questions. Data collection and analysis for two experimental 
test versions are described in detail. Item and test statistics for 
the final version were collected from four southern Ontario high 
schools and compared to results of two other critical thinking tests. 
The appendices contain two versions of the Test on Appraising 
Observations; the Test on Assessing the Believabili ty of Observation 
Statements; the Observation Test Interview Model, B; the Instruction 
Sheet to Cooperating Teachers; and the answer key and principles 
tested per item. (BS) 
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Chapter .1 
Introduction 

• » 

This report describes the design of a critical thinking test of high 
school students' ability to appraise observations. Statements of 
observation, as well as fundamental principles of value, are 1 at the 
foundation of much of what we believe. If critical thinking is conceived as 
dealing with decisions about what to believe, as it is by Robert Ennis 1 and ' 
David Hitchcock, 2 then basic to such decisions will be appraisals of the 
confidence to place in particular' statements of observation. This 
• recognition, along with the fact that there is no readily-available test of 
this aspect of critical thinking ability, motivated this project. 

In addition to the obvious outcome, of a published test and manual, 
the project had two other major objectives. The first was the elucidation 
and defense of a set of principles upon which to base appraisals of ( 
observations. This has been accomplished and reported. 3 The second was 
the elaboration and the testing of a methodology of gaining test validation 
information by probing people's reasons for responding to test questions the 
way they do. The methodology is explained in this report, but the 



Ennis. R.H. A concept of critical thinking, Harvard Educational Review, SS, 1962 
81-111. 

■> 

"Hitchcock, D. Critical thinking: A guide to evaluating information, Toronto: 
Methuen, 1983. 

N'orris, S.P. Defining observational competence, Science Education, 63 1981 
129-142. 
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employment of the technique led to further unanswered questions which 
are currently being explored. In brief, I am attempting to determine to 
wljat extent people's verbal reports of their thinking on tests are accurate 
reflections of the thinking which takes place, and to what extent the 
manner of eliciting those reports influence the reports' contents. This 
study is yielding interesting results and will be reported in the near future. 

1.1. The Perceived Need for Critical Thinking Research 

Critical thinking hasjong been held in high esteem by educators and 
educational theorists. However, it is only recently that there has been 
support for a concerted effort to implement critical thinking instruction 
into grade school and university. The result has been the introduction of 
critical thinking materials (although they are not always called 'critical 
thinking') at all educational levels, and a growing network of 
communication among those interested in teaching, and theorizing about 
critical thinking. For example, the primary and elementary grades have 
seen the development of a philosophy for children course, which has had 
increasing popularity throughout the world. One unit of this programme, 
Jlarry Stottlemeier's Discovery* concentrates on teaching children how to 
think philosophically. In addition, the institute which developed this 
programme publishes a journal, Thinking, devoted to theoretical and 
practical problems in teaching critical thinking. 

At California State University critical thinking is being taken 
seriously to the point where the Chancellor of the University issued an 
order, Executive order 338, making the study of critical thinking a 
requirement for every undergraduate in the system. One result of this 
move has been the initiation of a critical thinking newsletter, CT News, to 
facili te communication among those interested in teaching critical 

h.ipman, M. Harry Stottlemeier's discovery. Upper Montclair, New Jersey: The 
Institute for the Advancement of Philosophy for Children, 1977. 
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thinking, and the regular sponsorship of conferences on critical thinking in 
California. t A 

In Canada, the current thrust in critical thinking is primarily at the 
university level, and involving philosophers and philosophers of education. 
There have been two books on the subject published in Canada in the past 
couple of years. One is by Hitchcock, mentioned previously, which is a 
text book on critical thinking methods. The other, Critical Thinking and 
Education, 5 is mainly a theoretical analysis of the concept of critical 
thinking and of the place of critical thinking instruction in education. 

• 

Two philosophers at the University of Windsor are responsible for .the 
birth of a new journal devoted to critical thinking, Informal Logic. They 
have also sponsored two international conferences on critical thinking, or 
informal logic as it is sometimes called, which have in turn led to the 
institution of a new society, The Association for Informal Logic and 
Critical Thinking, formed to promote interest and dialogue in this area. 

It seems, then, that there is^ a growing interest in critical thinking 
instruction which goes beyond the mere lip service that has often been paid 
to this goal of education. However, the increased interest has highlighted a 
number of serious deficiencies. Much work is needed on the 
conceptualization r what constitutes good critical thinking, on the 
development and testing of approaches and materials for instruction, and 
on designing additional techniques for assessing the extent to which people 
are critical thinkers. In the following chapter there is a discussion of the 
ivailable critical thinking tests and of the existing analyses of critical 
thinking. 



8 5 

7 McPeck, J.E. Critical thinking and tducation. Oxford: Martin Robertson. 1981 
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Chapter 2 

Critical Thinking Testing 
and Conceptualization 

The way in which one conceives of critical thinking will affect how 
one categorizes attempts to test for it. If one thinks of critical thinking 
ability as a multi-faceted competence as J do and as Robert Erfnis does, 6 
then one tends to see two types of critical thinking te3ts, general ones and 
aspect-specific ones. General critical thinking tests are those which 
attempt to give an indication of people 's critical thinking ability in general. 
That is, they test for many or all aspects df critical thinking. Aspect- 
specific critical thinking tests are those,, which (attempt to test for only one 
aspect of critical thinking, such as inductive thinking ability, deductive 
thinking ability, assumption-finding ability, observation appraisal ability, 
or whatever the case may be. There are advantages and disadvantages to 
both types of tests. One gives broad coverage but virtually no detailed 
knowledge or diagnostic assistance. The other type gives detailed 
knowledge and diagnostic assistance but at the expense of a narrow focus. 
The test of observation appraisal designed in this study is an aspect-specific 
critical thinking test. 
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2.1. Critical Thinking Tests 

At present there are five, general, easily scoreable, English language 
critical thinking tests readily available in North America. .These are the 
Cornell Critical Thinking Test, Level A', 7 the Cornell Critical Thinking 
Test, Level Z* the New Jersey Test of Reasoning Skills, 0 the Ross Test 
of Higher Cognitive Processes, 10 and the Watson-G laser Critical 
Thinking Appraisal. These tests purport to provide an easily usable way 
of acquiring a br^ad picture of people's critical thinking competence, 
although each of the tests suffers some flaws. 12 

• 

The situation concerning aspect-specific tests of critical thinking is in 
far worse shape. In an extensive review of the literature on testing 'critical 
thinking Bruce Stewart 13 discovered several unpublished general critical 
thinking tests, and also several tests of deductive thinking ability, one 
aspect of critical thinking ability. However, he found few tests of other 
aspects of critical thinking, a- glaring deficiency if one educational goal is to 
acquire detailed diagnostic information of students critical thinking, ability. 



'Ennis, R.H. and Millman, J. Cornell Critical Thinking Test, Level X. Champaign, 
Illinois: The Illinois Thinking Project, 1982a. 

g 

Ennis, R.H. and Millman, J. Cornell Critical Thinking Teat; Level Z. Champaign, 
Illinois: The Illinois Thinking Projeat, 1982b. , 

9 Shipman, V. New Jersey %est of Reasoning Skftls. Upper Montclair, New Jersey: 
Institute for the Advancement of Philosophy for Children, 1983. 

10 Ross, J.D. and Ross, CM. Ross Test of Higher Cognitive Processes. Novate 
California: Academic Therapy Publications. 1976. 

* l Watson, G. and Glaser, E.M. Watson-Glaser Critical Thinking Appraisal. New 
York: The Psychological Corporation, 19§0. 

"Ennis, R.H. Problems in testing informal logic critical thinking reasoning ability. 
Informal Logic, 6 t 3-9, 1984. 

13 

Stewart, B.L. Testing for critical thinking: A review of the resources. (Rational 
Thinking Reports Number 2). Urbana, Illinois: The T Uinois Rational Thinking Project, 
1979' 

\ 
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Regarding observation appraisal Stewart discovered one test, 
Recognizing Reliable Observations, developed by the Instructional 
Objectives Exchange (IOX). 14 This test is based upon a set of principles of 
^ob^rvation appraisal developed by Robert Ennis. We have modified this 
same set of principles for the observation test developed in this study. 
However, for the IOX test no validity or reliability information, other than 
a content validity judgement by expert opinion i js_£r^s£Jited4^ 
Also Stetoartjw^ confirms, that many of the 

reported observation statements on the test are not observation statements 
at all, but statements of inferences. 16 Another problem is that the test 
contains only ten multiple-choice questions meaning that the principles for 
judging observation statements, which . ffr outnumber ten, are poorly 

i 

covered, and that the test is likely to have poor reliability. 

Others have explored the measurement of critical thinking ability, 
but their work has not received wide.distribution nor are the tests which 
w»>re developed readily available. In this regard work by D.P. Wright 17 
and by the members of the Wisconsin Research and Development Center 



Instructional Objectives Exchange. Recognizing Reliable Observations. Los Angeles: 
Instructional Objectives Exchange, 1971. 

15 Ibid., p. 104. 
\ 6 

I have discussed the distinction between observation and inference in the following: 
A concept of observation statements. In D R. DeNicola (Ed.) Philosophy of Education 
1931, Normal, Illinois: The Philosophy of Education Society, 1982; A speech act 
conception pf observation statements. In S. Clarke and R. King, Papers from the sixth 
annual meeting of the Atlantic Provinces Linguistic Association, St. John's, 
Newfoundland: Memorial University of Newfoundland, 1982; Defining observational 
competence, Science Education, 68, 1984, 129-142; Observation in science and science 
education, Paper given at a conference on Heuristics in Mathematics and Science 
Education, Iqstitute for Logic and Cognitive Studies, University of Houston, July, 1984. 

17 Wright, DP. Instruction in critical thinking: A three part investigation. Paper 
presented at the annual meeting of the American Educational Research Association, New 
York, 1977, ERIC Document Reproductioa-Service No. ED 138 518. 
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for Cognitive Learning are important to mention because both employed 
principles of- sound thinking in constructing the theoretical foundation for 
their wdrk. This-is,.the approach we have adopted also, since explicitly 
stating' one's principles for good thinking gurdes test construction, 
curriculum development, and theorizing. In my mind, however, both 
_pieces-of-work-b^v«-serio«slktws whiefr the current study avoids. 

Wright's work runs the risk of encouraging students to believe that 
the principles of thinking are exceptionless formulas for" getting answers, 
and (hat judgements can be made when little of the relevant information is 
at hand. The work does this by requiring students to make judgements in 
situations in which no context is supplied. For example, one of Wright's 
questions is designed to assess whether or not students act in accord with 
the following conflict of interest principle: you get more reliable 
information from a person who has nothing to gain from his or her 
statement being correct than from a person who does. The students are to 
judge that Mrs. Truman's statement in the following example is reliable 

because she has no conflict of interest. 

Mrs. Truman teaches fifth grade. One day two fourth grade 
boys argued about who owned a ball. Mrs. Truman listened to 
Ted and Don and said, "I think it's Don's. - 

Given the small bit of information provided in the description of the 
situation a careful student would probably not conclude that Mrs. 
Truman's statement is reliable, and justifiably so. The most defensible 
response to this question would be to withhold judgement because not 
enough is known about Mrs. Truman's motives for ruling in Don's favour. 
We have attempted to avoid this problem in the test described in this 
study by setting the questions in the contexts of stories which provide the 
information required for making informed choices. 

18 

Allen. R.R.. Feezel, J.D.. and Kauffeld, F.S. A taxonomy of concepts and critical 
abilities related to the evaluation of verbal arguments. Occasional paper no. 9. Madison, 
Wisconsin: Wisconsin Research and Development Center for Cognitive Learning. The 
t'niversity of Wisconsin, 1967. Eric Document Reproduction Service No. ED 016 658. 
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There are two major problems with the work done by the members 
of the Wisconsin Research and Development Center for Cognitive 
Learning: (i) the set of principles of sound thinking, consisting of only 
eight, is not comprehensive and (ii) some of the principles are stated as 
necessar conditions for reporting observations well, although they are not 
necessary conditions. For example, one of the principles says that in order 
to report observations well a person must be relatively free of bias or 
concern for personal benefit. 19 This is not so, however, since other factors 
such as the person's concern with reporting honestly might counteract the 
tendency to bias the report for personal gain. The principle would have 
been better stated had it said that a person's being likely to personally gain 
from his or her statement being correct tends to make that person's 
statement unreliable. 

2.2. Critical Thinking Conceptualization 

Much of the discussion of the previous section indicates that sound 
critical thinking test cons rucriou and test response interpretation depend 
upon having a comprehensive and sound analysis of the nature of good 
critical thinking. T one's analysis is not comprehensive, one obtains only a 
partial picture of people's critical thinking competence. If one's analysis is 
not sound, there is the risk of falsely categorizing those who are not good 
critical thinkers as being so, and those who are as not being so. In this 
study we have relied extensively upon Robert Ennis' analysis of critical 
thinking because in my estimation it is both sound and reasonably 
comprehensive, but also because it " is in terms of principles of good 
thinking. 

The Ennis approach of defining critical thinking in terms of guiding 
principles of thought has some distinct advantages. Consider the following 



19 Ibid.. p. 17. 
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principle as an example: "An observation statement tends to be believable 
to the extent that the observer has no conflict of interest." This principle, 
or more precisely the whole set of such principles of which this is just 
one," has at least three practical uses. First, it provides as precisely as *he 
subject matter will allow guidance for evaluating reports of observations. 
It says that the believability of such reports is diminished to the extent 
that the observers are in a conflict of interest. Note that this does not 
mean that being in a conflict of interest makes it impossible for an observer 
to make a credible report, as implied by the principle supplied by the 
Wisconsin Research and Development Center for Cognitive Learning. Thi§* 
would be too strict. It also does not provide any clear formula to indicate 
by how much conflict of interest reduces believability. Agai", this is the 
only appropriate stance, since all such judgements of degree must be made 
in the context of some situation, taking into account all the information 
that is available. The principle, says that all other things being equal the 
person in a conflict of interest is less credible than one who is not, and to 
this extent its guidance to evaluation is clear. 

The second practical use of this principle, and all the others which go 
with it, is that they provide a core for a curriculum designed to improve 
students' thinking in this area. A presentation of the principles, a 
discussion of their meaning and intended mode of use, and provision for 
practice in applying the principles to the evaluation of statements of 
observation is the outline for a course on this aspect of critical thinking. 

Thirdly, the set of principles provides the framework for designing a 
test of students* ability in this area, because it provides the justification for 
choosing correct answers on the test and a standard against which to judge 
the completeness of the test. The principles can be used for justifying test 



The whole set of principles and a discussion of them will appear in the following 
chapter. 
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answers because in a test one can effect the sort of control over the 
situation implied by the "All other things being equal" expression used 
above. That is, in a test one is able to contrive a situation in which all 
factors save one, say the observers' conflict of interest, are controlled. If 
one thinks of test items as experimental set-ups, and the giving of an item 
as running a trial of an experiment, as some do, 21 then all of this makes a 
good deal of sense. 

The above considerations motivated me to base the test construction 
upon Ennis' t conception of good observation appraisal. However, an 
examination of his set of principles in light of work in the philosophy of 
science and experimental studies of eyewitness testimony suggested that 
Ennis' set of principles needed some modification. This reanalysis was the 
first priority, before a draft test could be prepared. The resulting set of 
principles of appraisal is described in Chapter 3. In addition to these 
considerations, the nature of the study was influenced considerably by an 
attempt to use a long-touted but seldom employed method of test 
validation which relied on studying the mentar- processes of people while 
they responded to questions on the test. The motivation for adopting this 
approachyand the exact form it took on will be described in Chapter 4. 



Tomko. T.N. The logic of criterion-referenced testing. Unpublished doctoral 
dissertation, University of Illinois at Urbana-Champaign, 1981. 
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Chapter 3 

Principles for 

Appraising Observations 

1 

Public reports of observations are often subject to appraisal. 
Scientists must submit their observations to the examination of other 
scientists, eyewitnesses must submit their testimony to the evaluation of 
jurors. Assessing such reports of observations is a complex task, and there 
cau be no exact formulas for how to do it properly. However, the greater 
orie/s knowledge of the factors which affect the accuracy of observation 
reports the greater one's ability to make sound assessments. \^hen there 
are factors which influence reporting accuracy in a systematic way, then 
there is the possibility of codifying these effects and producing a set of 
principles of assessment based on their systematic occurrence. Table 3-1 
contains the most comprehensive set of such principles that we have been 
able to produce thus far. 

There are several things to notice about the set of principles. First, 
there are four categories. Principle I compares the believability of reports 
of observations to the believability of inferences based upon them. 
Principles II, ffl, and IV relate believability to characteristics of the 
observer, observation conditions, and the observation statement itself. 
Each of the latter three principles consists of several subprinciples, each 
addressed to a particular factor. 

The principles must be interpreted cautiously. Specifically, it is not 
proper to treat them severally as either necessary or sufficient conditions 
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Table 3-1: Principles for Appraising Observations 



!• Observation statements s tend to be more 
believable 'nan inferences based upon them 



II. An observation statement tends to be believable 
to the extent that the observer 

1 is functioning at a moderate level of emotional 
arousal. 

2 is alert to the situation and gives his or her 
statement careful consideration, 

3 has no conflict of interest. 

4 is skilled at observing the sort of thing observed. 

5 has a theoretical understanding of the thing 
observed. 

6 has senses that function normally, 

1' has a reputation for being honest and correct. 

8 uses as precise a technique as is appropriate. 

9 is skilled in the technique being used. 

1 0 has no preconceived notions about the way the 
observation will turn out. 

1 1 was not exposed, after the event, to further 
information relevant to describing it: 

(If the observer was exposed to such information, 
the statement is believable to the extent that the 
exposure took place close to the time of the event 
described ) 

1 2 is mature • 



III. An observation statement tends to be believable 
to the extent that the observation conditions 

1 provide a satisfactory medium of observation, 

2 provide sufficient time for observation. 

3 provide more than one opportunity to observe, 

4 provide adequate instrumentation, if instru- 
mentation is used 

(If instrumentation is used m gaming access. 

then the statement tends to be believable to the 

extent that the instrumentation 

a has suitable precision. 

b has a suitable range of application. 

r. >s of good quality. 

d works in a way that is well understood. 
♦* «s »n good working condition ) - 



IV. An observation statement tends to be believable 
to the extent that the observation statement: 

1 . commits the speaker to holdmrj a small number 
of things to be true. 

2 is corroborated; 

3. is no more precise than can be justified by the 
observation technique being used, 

4 is made close to the time of observing; 

5 is made by the person who did the observing. 

6 is strongly believed to be corroboratable by the 
person making it. 

7. does not conflict with other statementsfor which 
good reasons can be given, 

8 is made in the same environment as the one in 
which the observation was made; 

9. is not about an emotionally-loaded event. 4 

10. is the first report of the event provided by the v 
speaker; 

1 1 is not given m response to a leading question. 

12. does not report a recollection of something 
previously forgotten; 

i 

13. reports on salient features of an event, 
(Features of an event are salient to the extent 
that they are extraordinary, colourful, novel, 
unusual, and interesting, and not salient to the 
extent that they are routine, commonplace and 
insignificant.) 

14 is based upon a reliable record, if it isbased upon 
a record 

(If an observation statement is based upon a 

record, then the statement tends to be believable 

to the extent that the record 

a was made close to the time of observing, 

b was made by the person who did the 

observing. 

c comes from a source having a good reputation 
for making correct records ) 
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for observation statements to be believable. That is, one or another of the 
conditions may not be satisified on a particular occasion but still a person 
may observe well. In addition, although one or another of the conditions 
may be satisified on a particular occasion, a person may still observe 
poorly. Thus, the expressions "tends to be believable to the extent that" 
and "tend to be more believable" used in stating the principles are meant 
to be taken seriously. They are qualifiers which indicate the limitations of 
the principles p id stress that they are not exceptionless formulas. They 
must be applied judiciously, taking into account the characteristics of the 
situation at hand and relevant background knowledge, including experience 
in related matters. Application of the principles to actual cases is not a 
trivial matter, and requires an ability over and above knowledge and 
comprehension of the principles themselves. For example, in actual 
situations the principles often compete, pushing one's judgement of 
believability in different directions. In the case of two observers with 
different levels of access to the phenomenon, for instance, the one with the 
better access might also be the one with the lesser skill. Thus, an aporaiser 
Jyvould have to weigh the different factors, taking into account the amount 
of skill needed for the phenomenon being observed and the comparative 
degrees of access which the observers have to the phenomenon. One 
might, depending on the situation, decide that good observational access is 
more important than high level of skill and conclude in favour of the 
observer with the better access. On the Other hand, in a different situation 
the need for observational skill may far outweigh the need for good access, 
and an appraiser's judgement should change accordingly. 
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3.1* Justifying the Principles 

Many of these principles are readily acknowledged by experienced, 
thoughtful people, and in that sense may seem trivial. However, research 
on eyewitness testimony appraisal 22 indicates that, in general, typical 
adults appraise observation reports using unsound principles. For example, 
jurors tend to evaluate eyewitness testimony according to such factors as 
the agreeableness of witnesses on the stand, their dress and physical. 

bearing, their having cultured speech, and their expressing confidence in 

o 

the truth of what they say. In addition, my own research on high school 
students" has revealed that they make consistent errors of reasoning, such 
as accepting things at face value when this is not warranted, relying 
unconditionally on the words of experts, and placing too much trust in the 
confidence which speakers display and in the definite-sounding manner of 
their statements. So while the principles may seem, trivial to some, there 
are many people, including adults, who have very little knowledge of them. 

The above argument points to the need for instruction in some 
principles of sound appraisal, but there remains the question of whether or 
not the particular principles in Table 3-1 are valid. Nelson Goodman 24 

once argued that principles of inference, of which the principles in Table 

» 

3-1 are an instance, and particular inferences which we make are mutually 
justified by being brought into agreement with one another. That is, there 

"Lindsay, R.C.L., Wells, G.L. and Rumpel, CM. Can people detect eyewitness- 
identification accuracy within and across situations? Journal of Applied Psychology, 66 t 
1981, 79-89; Loftus, E.F. Eyewitness testimony. Cambridge, Mass.: Harvard University 
Press, 1979; Wells, G L., Ferguson, T J. and Lindsay, R.C.L. The tractability of 
eyewitness confidence and its implications for triers of fact. Journal of Applied 
Psychology, 66, 1981, 688-696; Yarmey, AD. The psychology of eyewitness testimony. 
New York; The Free Press, 1979. 

♦ 

Norris, S.P. and King, R, Observational ability: Determining and extending its 
presence. Informal Logic, forthcoming. 

Goodman, N. Fact, fiction, and forecast. Second Kdition. Indianapolis: Bobbs- 
Merrill, 1965. 
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is an interplay between the two, each having the possibility to influence the 
other. He claimed: "A rule is amended if it yields an inference we are 
unwilling to accept; an inference is rejected if it violates a rule we are 
unwilling to amend". 26 The "we" in Goodman's directive is of course 
bothersome. If the "we" refers to the adult population at large, then the 
studies of eyewitness testimony mentioned above indicate that many 
patently false inference rules and inferences would have to be sanctioned. 
In 'addition, Stephen Stich and Ricnard Nisbett 26 have shown that this 
conclusion pertains to more than inferences about eyewitness testimony. 
Large- numbers of the adult population make many sorts of unjustified 
inferences. These include improper assignments of probabilities to events 
(such as believing that the chances of flipping a head are increased after 
many tails are flipped successively), failing to account for statistical 
regression, and improper conclusions of causation f*om correlational data. 
Interpreting "we" broadly makes Goodman's rule dangerous, and thus for 
our purposes we modified the rule 

Stich and Nisbett suggested that Goodman's rule couid be made 

usable by replacing the "we" with "the socially, consensugljy, designated 
• • 27 

lutnonties". Part of the justification for our principles is provided by 
using Goodman's rule modified in this way. Many of the principles in 
Table 3-1 are based upon judicial practice. Rules of inference and of the 
admissibility of evidence have evolved in that field through the mi'tual 
adaptation of rules and particular inferences. Principles such as those 
dealing with conflict of interest, observer expertise and reputation for 
veracity, degree ct access to the phenomena observed, leading questions, 

.1 

Jr> Ibi(i., p. 64 I 
•jg \ 

Stich, S.P. and Niibett, R E. Justification and the psychology of human reasoning. 
Philosophy of Science, V. 1980, 188-202. 

~' Ibid., p. 201 
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and record making, are examples of the type meant. If the judiciary is 
taken to be an appropriate, socially recognized authority in the sense 
above, then the use of these principles by the courts lends them credibility. 
In constructing the original set of principles upon which the one here is 
based, Robert Ennis 28 took a similar stance of relying on the practice of 
the courts. 

Judicial practice is not, however, the only source of support for the 
principles in Table 3-1. In addition, there has been considerable research 
into the factors which affect the accuracy of eyewitness testimony, and 
generalizations about these effects can serve as the basis for principles. For 
example, this research consistently shows that witnesses report less 
accurately on emotionally-loaded events, ones in which there is violence, 
say, than on emotionally-neutral events. 29 Given the weight of the 
evidence we think it justified to include Principle IV.9: An observation 
statement tends to be believable to the extent that it is not about an 
emotionally-loaded event. Other principles supported by research from 
this field include those about reporting on salient features of an event, and 
being exposed to information relevant to the description of an event after 
the event has occur* ed. The principles are IV. 13 and 11.11, respectively. In 
these aspects of observation assessment, psychological research provides a 
supplement to judical practice. Even more strongly, the research provides 
ground 6 * for changing certain features of judicial practice, and inferential 
practice in general. The role extends beyond a mere description of the 
types of errors of inference and appraisal which people make, into 



~ Knnis. R ll. A concept of critical thinking, harvard Educational Review, 1902, 
81-111. 

Loft us, Ibid. 

'^Vor example fhe Law Reform Commission of Canada has made use of this research 
^ee. Brooks. N. F > olice guidelines: Pretrial eyewitness identification procedures. Ottawa: 
Law Reform Commission of Canada, 1983. 
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providing rather direct information on how people ought to reason, and 
into how practice ought to proceed. For example, some psychojogists are 
now using the results of their research to try to convince the judicial 
system that certain traditional modes of witness interrogation is likely to 
be counterproductive. Principle IV. 10 provides a case in point. It is now 
common practice that witnesses must give accounts of their testimony on 
— x many occasions, to police officers, to lawyers, to insurance agents, and to 
the courts. However, evidence from eyewitness testimony research shows 
that people's accounts of what they have witnessed become less accurate as 
they give more and more reports. The accuracy is reduced to a larger 
degree than can be accounted for merely by removal in time from the 
event. Thus, there is reason to try to reduce the number of accounts of 
their stories which witnesses are asked to relate. 

A third source of support for the principles derives from our 
common-sense psychology. Of course, one must be careful not to place 
undue emphasis on common-sense ideas, and be prepared to change them 
in the face of more systematic evidence. However, many of the principles 
are very plausible in light of common-sense notions and derive some of 
their support that way. For example, the principles about observer 
alertness, skill, and theorei»oal understanding, as well as those about 
adequate time and conditions for observing, derive partial support in this 
wajr. v In each case we pl-'isibiy think of there being an accuracy-reducing 
mechanism in operation. Thus, we plausibly believe that if a person is not 
attending to an event then that person is less believable, because our 
common sense tells us that without attention memory traces of events are 
not stored or are stored inaccurately. 

It is best, then, to think of the principles as being supported by a 
network of information. The practice of the courts, evidence from research 
<>n eyewitness testimony, and common-sense notions of the psychology of 
human beings serve jointly as both their source and their support. The 
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case is not the fin^l word. New research and new understandings will 
likely lead td modifications. However, for now the set of principles is the 
most comprehensive and most accurate one available. 
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Chapter 4 
Test Validation 

The theory of test validation is a theory of how one ought to justify 
certain uses to which tests are to be put. If the use is as a measure of some 
psychological trait or construct, such as a mental ability, then the relevant 
validation technique is construct validation. The theory of construct 
validation is a theory of how to justify tests as measures of psychological 
traits. I take the trouble to say Jl this, and to risk repeating the obvious 
for many readers, for two reasons. First, construct validation played a 
central role in this study because the attempt was to produce a justified 
measure of a mental ability, the ability to correctly appraise observations. 
Second, the techniques of construct validation used in this study 
emphasized some approaches which are typically deemphasized in test 
validation studies, and placed less stress on those typically viewed as most 
important-. Since this is so, this chapter is included to show why the 
approach tu^en here was adopted. 

4.1. The Basis of a Theory of Construct Validation N 

I have argued elsewhere 31 that any theory of construct validation 
retfs fundamentally on a theory of psychological constructs, or to put it 
another way, on a theory of the role of theoretical terms in science. None 
of this sh(Jtitd be surprising. Psychological constructs are theoretical terms 



Norris. S P. The inconsistencies at the foundation of construct validation theory. In 
K R. House (Ed.), Philosophy of Evaluation. New directions for program evaluation, no. 
V}. San Francisco; Jossey-Bass, 1983, 
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in the same sense as terms such as "electron", "black hole", "gene", and 
our understanding of what we mean by these terms has to be fundamental 
to our conception of how science proceeds and what science produces. 
Specifically, a theory of theoretical constructs will determine to a la / fg6 
extent views of the nature of causal explanation, of.the conception of truth, 
of standards of adequacy for judging theories, and of scientific theories 
themselves. 

/ 

The problem that I perceive with construct validation theory as it is 
portrayed in the educational and psychological testing literature is that it 
is motivated by two inconsistent views of the nature of theoretical 
constructs, resulting in inconsistent views of causal explanation, of truth, of 
standards for judging theories, and of scientific theories. These 
inconsistent views are explicitly expressed in the literature, but they are 
not acknowledged. The result is that in essence there are two views of 
construct validation to be found in the test theory literature, though they 
are presented as one. Without giving textual support for the existence of 
these two views in the construct validation literature 32 I will outline the 
two views, and indicate how the validation methodology employed in this 

study is a result of adopting one of the views rather than the other. 

V. 

4.1.1. Theoretical Terms 

Theoretical terms in science have generally been interpreted in one of 
two ways. One of these, finding its roots in the work of David Hume 33 and 



30 

"This support ran be found in t a° article by me mentioned above and also in Norris, 
S.P. A pitfall in the construct validation of ability tests, Unpublished doctoral 
dissertation, University of Illinois at Urbana-Champaign, 1981; and Tomko, T.N. The 
logic of critierion-referenced testing, Unpublished doctoral dissertation, University of 
Illinois at Urbana-Champaign, 1981. 

Hume, D. An inquiry concerning human understanding. C.VV. Hendel (Ed.). 
Indianapolis. Indiana. Bobbs-Merrill, 1955. Originally published, 1748. 
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the philosophy of logical positivism expounded by The Vienna Circle, 34 
views theoretical terms instrumentally. That is, they are useful 
instruments, useful for categorizing the observable world and relating the 
subsequently-pfoduced categories to one another. The intended 
implication is that theoretical terms and the constructs to which they refer 
have no tie to the external world, except as ideas in the minds of scientists, 
that is. r 

In contrast to this positivistic view of theoretical terms, there is a 
view which maintains that the significance of theoretical terms in science is 
fundamentally no different from that of the terms we use for referring to 
everyday things, such as "chair,", "water", and "grass". Part of the 
significance of these everyday terms is that we take them to refer to real, 
things, in the world: chairs, water, and grass in the cases above. 1 To 
interpret scientific terms in this way is to assume that there is something 
real to which they also refer, although those real things are usually not 
directly observable, and often not likely to ever be. So, for example, to 
interpret the theoretical term "electron" in this way is to maintain a 
position much like the following one expressed by Hilary Putnam: 35 

The statement that there are electrons flowing through a wire 
may be as objectively true as the statement that there is a chair 
in this room or the statement that I have a headache. Electrons 
exist in every sense in which chairs (or sensations) exist; electron ^ 
talk is no more derived talk about sensations or "observable 
things" than talk about sensations or chairs is derived talk about 
electrons. 

Applying these views to psychological constructs, such as ability 
terms, we find one view which maintains that abilities are no more than 



The Vienna Circle, [The scientific conception of the world}. In M. Neurath and R.S. 
Cohen (Eds.), Otto Neurath: Empiricism and sociology. Dordrecht, Holland: D. Reidel, 
1973. Originally published 1929. 

35 

Putnam/ H. Three kinds of scientific realism. The Philosophical Quarterly, SS, 1982, 
195-200. 



* 24 

classes of behaviours, and another view which maintains that abilities are 
unobservable, yet real, properties of people which give rise to their 
observable behaviours. Adopting one of these views is logically, and as it 
turns out empirically, linked to adopting related views on the nature of 
causal explanation, of truth, of standards of adequacy for judging theories, 
and of theories themselves. 

4.1.2. Causal Explanation 

Associated with his idea that theoretical terms have no reference 
outside the minds of scientists, David Hume proposed a theory of causal 
explanation which has followers to this day. The view is that causal 
connections, as real connections between things in the physical world, do 
not exist. Instead, causal connections are merely connections in the 
imaginations of people, stimulated by their having seen classes of events 
regularly "conjoined". Thus, to say that two events are causally related is 
to make a statement about the conceptual framework of people, not about 
some connection in the world. Hume argues for this position quite 
unequivocally: 

1 The first time a man saw the communication of motion by 
impulse, as by the shock of two billiard balls, he could not 
pronounce that the one event was connected, but only that it was 
conjoined with the other. After he has observed several 
instances of this nature, he then pronounces them to be 
connected. What alteration has happened to give rise to this new 
idea of connection? Nothing but that now he feels these events 
to be connected in his imagination, and can readily foretell the 
existence of one from the appearance of the other. When we say, 
therefore, that one object is connected with another, we mean 
only that they have acquired a connection in our thought. 36 

In contrast, there are philosophers who maintain that causal 
connections refer to real connections between the things causally related. 



Ibid., p. 86. 
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Rom Harre argues that it is the task of science to discover the often 
unobservable, but nevertheless real, objects and processes which comprise 
these causal links. Contrary to Hume's view, Harre believes that causal 
connections must be more than conceptualizations in the minds of 
scientists, because a logical construction in the mind of a scientist cannot 
make things happen. And, Harre believes, whatever else is the nature of 
causes, they actually exert influence, they make things happen. 

When applied to theorizing about psychological events, such as 
people's performances on mental ability tests, these views produce two 
versions of psychological explanation. In one, explanations of performances 
on these tests is given in terms of those performances being related in a 
consistent manner (conjoined) with performances on other tests or in other 
sorts of situations. The attempt is to produce a nomological network 38 
which displays at least many of the relationships which investigators find 
pertinent. According to the other view, the attempt is to give explanations 
in terms of the mental mechanisms and processes which give rise to the test 
performances. The aim is to show what made the person perform as he or 
she did. 

J 

Often, th ese different sorts of explanations when given in sketches are 
not readily distinguishable. For example, it is not apparent from the 
explanatory statement, "The students' performaces were caused by their 
level of critical thinking ability," which sort of explanation is being 
assumed. There is a need to probe deeper to discover the sorts of support 
that is offered for the explanations. If the support is solely in terms of 
conjunctions of performances, the explanation is Humean. If the 
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justification is in terms of evidence of the mental processes and 
mechanisms which gave rise to the performances the explanation is of the 
type which Harre espouses. More will be said about this in the following 
section. 

4.1.3. Standards of Adequacy for Judging Theories 

For those who. hold a Humean view of causation, theories are tested 
by deriving from them predictions of observable events and checking to see 
whether these predictions are upheld. To the extent that the predictions 
hold, the theories are supported; to the extent that they do not hold, the 
theories loose support. This view of scientific progress is called 
hypothetico-deductivism because it maintains that theories are first offered 
as hypotheses and then tested by deducing from them observable 
predictions. 

While many who do not adhere to a Humean view of causation see a 
role for successful prediction in testing the adequacy of scientific theories, 
they realize that there must be other criteria of success. This is so since 
there are many theories which do not yield testable predictions. The 
theories of evolution and of plate tectonics, and archeoxogical theories 
which attempt to explain unearthed discoveries, all receive support 
primarily (if not solely) from their ability to explain previously puzzling 
phenomena. A person who sees scientific adequacy in this light, draws a 
distinction between the conditions for successful prediction and those for 
successful explanation. Those who follow Hume's ideas typically conflate 
these conditions. 39 

Hypothetico-deductivism in one form or another has received 



Hempel. C.G. Aspects of scientific explanation. New York: Free Press. 1965; Keat, 
R. and I'rry. J. Social theory as science. London: Routledge & Kegan Paul, 1975. 
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widespread endorsement in the test validation literature. 40 In the 

Standards for Educational and Psychological Tests 41 the following 

statement is made: 

In obtaining the information needed to establish construct 
validity, the investigator begins by formulating hypotheses about 
the characteristics of those who have high scores on the test in 
contrast to those who have low scores . . . Such hypotheses or 
theoretical formulations lead to certain predictions about how 
people at different score levels on the test will behave on certain 
other tests or in certain defined situations, (p. 30) 

The Standards then indicate that confirmed predictions reflect 
favourably on the judgement of a test's validity and disconfirmed ones 
reflect unfavourably. This is hypothetico-deductivism in its explicit form. 

4.2. Assumptions of the Validation Methodology Used in 
this Study 

Hypoftietico-deductivism was not the validation methodology 
employed in this study. When hypothetico-deductivism is appropriate at 
all (there are those who argue it is never appropriate) 42 it is so in highly 
advanced areas of science, when it is at least plausible that predictions can 
be deductively derived from theories. In the case of critical thinking 
ability, or the ability to appraise observations in particular, there is no 
available theory from which predictions can be derived deductively. This 



Campbell, D.T. and Fiske, D.W. Convergent and discriminant validation by the 
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1955, 281-302; Guion, R.M. On trinitarian doctrines of validity. Professional Psychology, 
It. 1980, 385-398; Messick, S. The standard problem. The American Psychologist, SO, 
1975, 955-966. 
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is possible only when the relevant variables to be controlled are known. In 
the field of critical thinking ability it is not known, nor is there a theory of, 
how performances on critical thinking tests ought to relate to other 
performances. For example, it is not known how critical thinking ability, 
or any of its specific aspects, correlate with abilities in various school 
subjects or with performances outside the context of schools. Similarly, it 
is not known how the aspects of critical thinking ability relate, or ought to 
relate, to each other. Thus, though there are principles for appraising 
observation reports which concern three areas (the observer, the 
observation conditions, and the observation statement itself) it is not 
known, nor is there much reason to believe at this time, that knowledge of 
principles in one of these areas is related in a particular manner to 
knowledge of principles in the other areas. < 

The fundamental belief guiding the validation methodology of this 
study is that the essence of providing an explanation of some phenomena, 
say performances on a test, is providing a model of a mechanism (broadly 
construed) through which the phenomena were produced. 43 The essence of 
validating that explanation is attempting to find ways to discover whether 
that mechanism is indeed operative. For example, Robert Sternberg 44 has 
a/model of a mechanism through which people solve analogies of the type 
found on the Miller Analogies Test. The mechanism is comprised of six 
components or steps. When one understands each of these components oue 
can see that if this mechanism were operative in a person, and if there 
were no countervailing mechanisms operating at the same time, then that 
person could solve analogy problems of the type in question. Sternberg's 
research is largely devoted to attempting to discover whether this 



' Hh:v kar. R. A realist theory of science. Sussex: Harvester Press, 1978; Harre, R. Thr 
principles of scientific thinking. Chicago: University of Chicago Press, 1970. 

"Sternberg, R.J. Intelligence, information processing, and analogical reasoning: Thf 
cmiponentxal analysis of human abilities. Hillsdale, New Jersey: Erlbaum, 1977. 
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mechanism is indeed operative in people who do well on the analogy items, 
and with showing that other proposed models are less adequate because 
they fail, for one reason or another, to provide a mechanism which, if 
operative, would explain performance on the analogy items. 

For the type of problems which examinees are presented on the Test 

on Appraising Observations I have proposed a mechanism which can lead 

to good performance. Each question on the test fits into an ongoing story. 

One of the stories is about a traffic accident and each question in this 

sequence presents two statements about the accident made by people who 

were involved or who were witnesses. The examinees are to pick which, if 

either, of the two statements they have more reason to believe at the time 

thejstatements are made. According to my theory of critical thinking, a 

critical thinker answering questions of this sort would approach them 

somewhat this way: 

The person would first of all look for relevant differences, 
either between the speakers (or if the same speaker is involved 
between that speaker's two occasions of speaking), between the • 
conditions under which the observations were made, or between 
' the types of statements that were made. If differences were 
found, their relevance would be judged according to whether or 
not there is some general tendency for such a difference to 
matter. For example, if one of the observers is a person who was 
involved in the accident and the other is a witness, then the 
examinee would note that the statement of the first is possibly 
influenced by a conflict of interest, and that people so influenced 
tend to be less believable than those who are not (all other 
relevant factors being equal). The examinee would then judge 
whether this general tendency concerning people in a conflict of 
interest applies in the particular situation, and if it does would 
reach the appropriate conclusion. 

The above description outlines a mechanism through which a 
particular mental ability, the ability to appraise observations, can operate. 
If indeed those who do well on the Test on Appraising Observations do so 
because their performances are governed by a mental process such as this, 
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and those who do poorly do so because they use some improper approach, 
then the test would be judged a valid test of the ability to appraise 
observations. It is possible of course that other legitimate approaches 
might be used which result in good performance. Such a result could mean 
different things, depending on the particular mental processes involved. 
The researcher would have to examine the processes and determine 
whether or not they are of a sort that might be due to critical thinking 
ability. If they are, then despite the fact that they differ from original 
preconceptions, the test should be judged valid. If however they are not 
(they might be judged as guessing processes, or as processes relying on 
unintended cues in the items), then the test should be judged invalid. 

AH of this suggests that in order to judge the validity of the test 
there has to be some attempt to access the mental processes which 
determine peof Vs performances on the test. The test is then judged valid 
to the extent that suitable mental processes lead to good performance and 
unsuitable ones lead to poor performance. The employment of a 
methodology for probing examinees' mental processes was at the heart of 
this study, and will be described in the following chapter. 
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Chapter 5 y 

The Evolution of the 
Test on Appraising Observations 

The Test on Appraising Observations in its present form (see 
Appendix A) is the result of an evolution through many versions. Each 
transition to a subsequent version was based on various considerations, 
some involving data which had been collected, others involving the best 
intuitions which we could muster. In this chapter, I describe in detail the 
basis of some of these transitions in order to illustrate more clearly the test 
development methodology that was used. Since what we were attempting 
was to provide at least a partial alternative to accepted test construction 

methodologies, it is important to give detailed descriptions. 

** y 

5.1. Decisions Concerning Audience and Style 

♦ 

5.1.1. Audience 

Before beginning any test construction one must have in mind (at 
least vaguely) the need which' the test is going to fill. As mentioned 
previously, I have perceived a lack of critical thinking tests of particular 
aspects of critical thinking ability. So, the first need to be filled by this 
test was for a critical thinking test of appraising observations. 

Beyond this decision, there are other matters related to whether the 
test is to serve classroom testing or research purposes (or both), or some 
other purpose. I decided to try to first fill the need of the classroom 
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teacher who often desires an easy to administer and score instrument, 
which can give rough indicators of students' attainment. Although .detailed 
knowledge of students' capabilities is desirable, it is not possible in current 
classroom settings to have a running account of what students are learning 
at the detailed level. Teachers need frequent feedback at the general level 
of the success of their instruction, and this is best provided with short tests 
(one class period or less) which require only a few minutes to score per 
examinee. 

Finally, one must decide on grade level. If one keeps the reading 
level of the test low, then one can produce a test which can be useful for a 
broad range of senior levels. For example, most tests which do not require 

A, 

the use of technical jargon .can be written at the reading level of the 

4 

average twelve or thirteen year old. So designed, theJest can then usually 
be used at the junior and senior high school levels, and often at the 
university level as well. We decided to produce a test which was suitable 
for at least the senior high school grades, since this is probably the most 
likely place that techniques of observation appraisal would be taught. In 
so doing, the Test on Appraising Observations is probably usable in both 
lower and higher grades, although up until now it has not been tried there. 

5.1.2. Style 

Most concisely, the test can be described as a multiple-choice test 
whose items are set in the context of stories, requiring one class period to 
complete, and designed primarily for the high school grades. However, this 
description is somewhat misleading, especially regarding the categorization 
as a multiple-choice test. "Multiple-choice" seems to be the best 
description, but we have taken steps to avoid some of the most serious 
pitfalls which tests of this type have faced in the past. So for those who 
tend to have an immediate adverse reaction to anything which might be 
called a "multiple-choice test", I urge you to consider how this test is 
different. 
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The test consists of two parts with items set in the contents of stories. 
Part A is the story of a traffic accident, and Part B involves the 
exploration of a river. The purpose of the stories is twofold. The primary 
one concerns the subject matter and the principles of observation appraisal 
which have been described. These principles are empirical generalizations, 
as was explained already. However, because they express tendencies and 
not universale,- they admit of exceptions. In order to decide whether it is 
appropriate to apply onejor another of the principles it is necessary to 
know the context in which it is to be used. This knowledge is necessary 
because extenuating circumstances can exist which can make application of 
a principle inappropriate. The contexts provided by the story lines in the 
test provide the required knowledge. Granted, they do not supply 
complete knowledge, which is tolerable since in no real situation is there 
ever complete knowledge. ^We must act on the best knowledge we can 
acquire. Providing spme knowledge of context, however, avoids passing on 
the impression that the principles of , observation appraisal can be 
mechanically applied in a knowledge vacuum. 

The other reason for' using stories is that they help to maintain 
interest in the test by keeping the examinees engaged in an evolving 
episode, in which one does not know what to expect next. We have found 
that this approach appears to have woN^d, because in informal 
conversations most students claim they did not find the test boring, 
although many said they found it puzzling. 

The test contains 50 items (28 in Part A). 45 Although we have 
always given the test as a power test, forty-five minutes of actual working 
time is sufficient for most senior high school students to finish. Each item 
presents the examinee with two statements in bold type. They are to 
choose which, if either, of the statements in bold type they have more 
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reason to believe at the time the statements are made. There are thus 
three alternatives 'for each item: (1) the J irst statement is more believable; 
(2) the second statement is more believable; and (3) neither is more 
believable, they are equally believable. In the sense that there are three 

i 

choices for each item, the test is a multiple-choice test. However, the 
choices for each question are the same, which is not usual for multiple- 
choice tests. This has both advantages and disadvantages. One 
disadvantage is that in having only three distractors per item, compared to 
the more usual four, there is an increased chance that random guessing will 
lead to correct responses, thereby reducing the validity of the test. 
However, we have found that guessing on this test is in fact a rare 
phenomenon, and we can turn to our interviews of examinees to support 
this. Answers were virtually always chosen for reasons among the 
examinees we interviewed, and performances for those not interviewed 
were virtually identical to the performances of those interviewed. Thus, 
there is little reason to think that lack of guessing among interviewed 
subjects was a phenomenon caused by the interview situation. 

Finally, though the number of distractors is usually greater than 
three for multiple-choice tests, when a test constructor is forced to find 
alternative distractors "to meet the required quota" there is a risk of 
having distractors of widely different attractiveness. 46 v The result is that 
the net number of distractors is smaller than the gross number, because 
examinees with only minimal knowledge of the subject matter can often 
eliminate at least one distractor because of its obvious implausibility. The 
three distractors used in the Test on Appraising Observations are the 
logically obvious ones, given the problems posed in the test, and no 
evidence was found of their being eliminated as implausible alternatives. 



Lord. F.M. and Novick, M.R. Statistical theories of mental test scores. Reading. 
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The final question of style concerns the nature of the problem as 
posed, namely, to judge for each question which, if either, statement in 
bold type is more believable at the time the statements are made. The 
decision to ask which is more believable was a deliberate attempt to avoid 
asking examinees to judge degrees of endorsement, such as would have 
been required had the distractors been, "strongly believe the first, strongly 
believe the second, weakly believe the first, weakly -believe the second, 
believe each equally," or some other similar set. /The motivation for doing 
this was the same that motivated Ennis in constructing the Cornell 
Critical' Thinking Tests. The problem is that "people with different 
levels of sophistication justifiably give different levels of endorsement to a 
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conclusion." Asking, however, for only the direction in which the 
evidence points is a means of obtaining higher agceement- among the best 
critical thinkers, and thus of more substantiated keyed answers. 

5.2. Interaction of Validation Methodology and Stages of 
Development 

A test construction is an exercise in design. Since this is so the 
designer's purposes and intentions play a fundamental role in the 
development and validation procedure. Although the designer need not be 
able to envisage all the uses to which an. instrument can or will be put, he 
or she must have some purpose in mind, and all unforeseen uses of the test 
will derive at least part of their justification from the quality which was 
originally built in. In this respect I take exception to a remark by 
L. J. Cronbach which has received considerable endorsement in the testing 
field, He said: 

The phrase validation of a test is a source of much 
misunderstanding. One validates, not a test, but an 
interpretation of data arising from a specified procedure. A 



Ennis, R.H. Problems in testing informal logic critical thinking reasoning ability. 
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single instrument is used in many different ways - Smith's 
reading test may be used to screen applicants for professional 
training, to plan remedial instruction in reading, to measure A he 
effectiveness of an instructional program, etc. Since each 
application is based on a different interpretation, the evidence 
that justifies one application may have little relevance to the 
next. 48 

I cannot imagine how all of this can be true. It is true as I have 
already said that any test can be used for different purposes, some not even 
foreseen when the test' was being designed. However, are all of these uses 
based on different interpretations, and is the evidence that justifies one 
interpretation irrelevant to the others? This is highly questionable. 
Consider the following example of a measuring device from the physical 
sciences. A voltmeter is designed to measur. inferences in electrical 
potential and to give va'id readings (within specified limits of error) when 
used in a specified way. Specifically, the voltmeter must be used with the 
type of current (alternating or direct) for which it was designed; it must be 
connected in parallel with the electrical resistance across which the 
potential difference is being determined; the electrical resistance of the 
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wires connecting the voltmeter to points in the circuit must be insignificant 
to the electrical resistance of the voltmeter itself; there must be no strong 
magnetic fields in the vicinity of the meter; and an unspecifid)ly large set 
of other conditions must be satisfied. Used in this way the voltmeter will 
measure voltage. However, just like Smith's reading test, the voltmeter 
may be used in many different ways. - For example,, it may be used to 
measure electrical current, or it may be used as an indicator of electrical 
circuit integrity. The voltmeter can be used to measure electrical current 
if the resistance across which it is connected is of known value. In such an 
instance, the known value of the resistance and the potential difference 
reading <>n the voltmeter can be used to calculate electrical current using 



Cronbach. L.J. Test validation. In R.L. Thorndike (Ed.). Educational measurement 
(^rond edition). Washington, D.C.: American Council on Education. 1971. p. 447. 
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Ohm's Law. which is a functional relationship among potential difference 

r ' 

resistance, and current. The voltmeter can be used to indicate circuit 
integrity, if the only way in which a potential difference reading is possible 
in the circumstances is that there is a complete circuit. In this latter case, 
the voltmeter is not used to measure anything, because any reading 
indicates circuit integrity. * 

The question is whether each of these applications is based on a 
different interpretation, and whether the evidence that justifies one 
interpretation is relevant to the evidence that justifies the others. The 
applications are, it seems to me, based on different interpretations of the 
voltmeter reading. In the first, the reading is interpreted as a measur- of 
potential difference; in the second, as a measure of electrical current; and 
in the third, as an indicator of circuit integrity. However, surely the 
evidence that supports the first application, the application for which the 
instrument was designed, is highly relevant to the two subsequent 
applications. In fact, the ouly reason that these subsequent applications 
are legitimate is that potential difference is functionally related to electrical 
cunent, and that to obtain a potential difference reading across a 
resistance there must be ^ complete electrical circuit present. That is, the 
subsequent applications and their justification are directly parasitic upon 
the intended application which guided the instrument design. There are. 
of course, uses to which the voltmeter can be put for which the intendel 
interpretation is completely irrelevant One might call to mind the high 
school physics student who, when asked to state how a barometer could be 
used to measure the height of a building, suggested as one answer that it 
be thrown off the roof and its time of descent determined! 

Much the same can be said for test construction and \ 'idation. The 
test constructor, or to emphasize the intentional aspects of construction, 
the test designer, has an intended interpretation which is to be placed on 
the scores of the test. The aim is to design a test for which this 
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interpretation is legitimate. The initial validation of the test is an 
appraisal of whether this aim has been reached/ If subsequently the test 
scores are to be interpreted differently from the originally intended 
interpretation, then the legitimacy of these interpretations must be 
determined. The original validation study will, however, be relevant in 
this regard, just as the validation of the voltmeter as a measure of voltage 
is relevant to its use as a measure of current or as an indicator of circuit 
integrity. 

The view guiding this study is that validity investigations of ability 
tests involve three steps: (i) finding out whether examinees understand the 
tasks on the test in the intended way; (ii) given that the appropriate 
understanding is there, determining whether examinees use appropriate 
approaches to complete the required tasks; and (iii) if number of correct 
answers is to be used as the indicator of examinees' ability, determining 
whether the ke>ed answers are justified given appropriate strategies for 
arriv at answers. A word on each step is in order. 

If examinees understand the tasks they are being asked to do ir * 
way different from that intended by the test constructor, then there is a 
risk >hat the intended interpretation of test scores will not be legitimate. 
Thus, if examinees taking the Test on Appraising Observations interpret 
ihe instruction to determine which statements, if either, are more 
believable as an instruction to determine which statements are true, then 
this might affect the legitimacy of interpretations of their scores. The 
situation is not straightforward. One would have to examine the basis for 
choosing answers. This examination might indicate that any interpretation 
of scores in terms of ability to judge comparative believability of reports of 
observations is ill-founded. On the other hand, the examination might 
reveal a discrepancy only at the word usage level, and not at the level of 
underlying meaning. 
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Given that the examinees understand the tasks in the intended way, 
it is not necessary that all examinees use the same approach to solve the 
tasks. The test designer may have an approach in mind, but must be open 
to there being other legitimate approaches to completing the same tasks. 
The appearance of such alternate approaches does not of itself mean that 
the test is invalid. However, since they cannot always be anticipated in 
advance (there is no way to predict' the approaches of ingenious examinees) 
such approaches must be examined for their legitimacy at the time they 
appear. / 

t 

f 
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Given the intended understanding and the use of appropriate 
approaches on the part' of examinees, it must be determined whether the 
keyed answer is appropriate. It is not unusual to find that examinees can 
follow approaches different from those anticipated by the test designer (yet 
appropriate all the same) to reach answers different from those keyed. The 
most usual result of this occurrence is the abandonment of the troublesome 
items. 

The general principle of ability test validation underlying this study 
is that ability tests are valid to the extent that good thinking leads to good 
performance on the test and poor thinking leads to poor performance. The 
attempt is thus to explain performance in terms of thinking, and to do this 
there must be a description of the thinking processes which lead to 
performance. It is important to note that this explanation need not be 
applicable to all examinees for the test to be suitable. It is because it is 
always possible, given our current lack of ability to identify and control the 
relevant variables, for good thinking to result in poor performance and for 
poor thinking to lead to good performance that the expression "to the 
extent that" was used above. For a test to be suitably valid, there must be 
at least an overall tendency for good and poor thinking to be iinked to 
good and poor performance respectively. 
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The approach to validating the Test on Appraising Observations was 
guided by the above general principle. Each stage of development was an 
attempt to bring the test closer to the ideal of having good thinking being 
the sole cause of good performance and poor thinking the sole cause of 
poor performance. Actually, the latter is more difficult for the test 
designer to influence, because poor performance can result from so many 
completely uncontrollable influences, poor attitude, lack of health, etc. 
The following section describes the stages of development of the test. 

5.3. Transitions Between Test Versions 

The current version of the Test on Appraising Observations is the 
result of the modification of several previous versions, some highly 
experimental which were used to chart rough directions, and some highly 
refined which required only cosmetic changes. I will describe each version, 
while concentrating on the most crucial points in the test's evolution. 

5.3.1. Preliminary Versions 

The earliest version of this test which was administered to a 
substantial number of subjects contained two parts and seventy items, 
fifty-nine in Part A. 49 There was an obvious imbalance in numbers of 
items in each part of the test which was subsequently altered. However, 
the test had taken on its basic form, with two story lines and three- 
alternative questions. 

This test was given to 51 sophomores in a public high school in 
central Illinois. The average score on the test was 37 items correct, with a 
KR-21 reliability of 0.75. Both of these results were judged respectable for 
a trial version, as were comments from students in discussions h,eld with 



Norris. S.P. Illinois Test on Assessing the Reliability of Observation Statements. 
Illinois Rational Thinking Test Series, Bureau of Educational Research, University of 
Illinois at Trbana-Champaign, July, 1979. 
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the classes afterwards that they found the test engaging.' However, certain 
obvious problems emerged. The instructions in this preliminary version 
told examinees to choose which, if either, statement in each question they 
found more dependable. Many students were not sure what was being 
requested, and attempts to solve their uncertainty with synonyms such as 
■reliable- and "trustworthy- did not seem to help. In the subsequent 
version the instructions told students to choose which, if either, statement 
they had more reason to bjelieve. This has proved satisfactory, and the 
instructions have remained the same in all versions since. 

Another obvious problem was that the test was too long. Students 
said that the test required them to work very iard, and near the end they 
were too tired to concentrate well. Also, only one-half the students 
finished the test and only three-quarters finished the first part. This was 
unacceptable for a power test. Test length was reduced to fifty items in 
the current version. 

There were also about a dozen items which had suspicious 
discrimination characteristics. These were items for which getting an item 
correct was negatively or negligibly related to overall performance on the 
test. Each of these items was examined and changes were made where the 
problems seemed to be occurring. 

A modified version of the test was then given to 94 public school 
students in another town in central Illinois. This version had not been 
shortened, and the effect was similar, with only about one-half the students 
finishing. However, the KR-21 was substantially higher, 0.85, and many of 
the items that formerly had suspicious discrimination indicies looked much 
better. However, there was still an obvious need of improvement. The test 
^4<mgth problem had to be corrected. In addition, the principles of 
ot^ervation were not at that time as widely based as they currently are. 
More work was required here, which led to a need for different items since 
new principles were added. 
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A final preliminary version was thus prepared which was based on an 
improved set of principles of appraisal (those in Table 3-1), which had a 
better balance in length between Parts A and B, and which was ,, v 
substantially shortened. The test contained 53 items in all with 25 in Part 
A. This version was administered to a small sample of 11 high school 
students from St. John's, Newfoundland, chosen for their availability 
during the summer months and given a small he Drarium for their 
assistance. Each student was asked to take just one part of the test (6 
took Part A) and was asked to think aloud while doing it. This procedure 
was beneficial in several ways. It indicated places where items or 
instructions were ambiguous, iud suggested how subsequent interviews 
might^be conducted. The first experimental version of the test, to be used 
to collect think aloud protocols from a large number of students, resulted 
from making modifications to this final preliminary version. 

5.3.2. Experimental versions 

\ The first experimental version of the test, called Test on Assessing 

the Delievability o f Observation Statements, Version B (?je Appendix B), 
was a 50 item test of the same format as the previous version. Three 
questions were dropped from the previous version, and 15 others were 
altered or replaced to accomodate the deficiencies which had become 
apparent in the 11 interviews and to effect a more even distribution of 
items across the set of principles. We were thus ready to collect the 
desired interview data. 

Our desire was to conduct the interviews in a fundamentally non- 
leading fashion. We wished to influence students' thinking as little as 
possible, realizing that just asking people to think aloud and placing them 
alone with a stranger might have effects in themselves. At the same time, 
it seemed that sometimes interrupting a student's narrative might be more 
beneficial than not, particularly when the interruption was merely to 
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clarify the ambiguous referent of a pronoun, or to point out obvious 
reading errors. In addition, although we did not wish to rush the 
examinees, to cut off reasoning by inadvertent signals, or to endorse or 
criticize particular reasoning attempts, we did wish to obtain as complete 
records of reasoning as were possible. To fulfill this aim it was often 
necessary to probe beyond the initial instruction to think aloud. This 
probing was done, however, only after examinees had chosen their answers 
to questions and had finished reporting on their thinking. Even in these 
follow-up stages, probing was as non-leading as possible, merely echoing 
examinees' already reported thoughts or asking them whether they could 
explain a little more about their choices of answers. The Observation Test 

Interview Model, B (Appendix C) indicates four stages^of the interview 

/ 

process/, ranging from the least leading to the most leading. The first stage 
merely, informed the examinees of the general purpose of the interviews 
and that they would be asked some questions while taking a test. 

Stage Two consisted in the first level of probe into examinees' 
thinking, and was non-leading. Examinees were asked only to tell all they 
could about what they were thinking as they were choosing their answers. 
This stage permitted interruptions only to probe for ambiguous references 
in examinees' reports, and for obvious reading errors. There was no 
provision for answering examinees' requests for additional information or 
for feedback on their progress. The interviewer was permitted to give only 
uninformative responses to su-h requests: "You can only go by what is 
written or "You can decide only according to what is said and what you 
know. ■ 

The third st.^e of interview was more leading and the particular 
probes used depended both upon the answers chosen and upon the thinking 
reports. We had developed a model of an ideal answer for eac'j question 
which involved first identifying the criterion or factor which made the 
difference in each case (one speaker was in a conflict of interest, one 
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speaker had more expertise, the observation conditions were better in one 
case than the other, etc.), then using this criterion to make a comparison 
between the two statements in the question on the basis of a general 
principle explaining the relevance of the criterion. If no identification of 
the criterion appeared in examinees' initial report, then the criterion was 
identified by the interviewer and the examinees were asked whether the 
criterion played any part in their thinking. If examinees mentioned the 
criterion but went no further in explaining the relevance of the criterion to 
their choice of answer, they were asked to tell (if they could) more about 
the difference the criterion made. For example, if in number 6 an 
examinee said that he or she chose the first statement (See Appendix B) 
because Ms. Vernon was a driver education instructor, then the student 
would be asked: "Could you tell me more about the difference Ms. 
Vernon's being a driver education instructor makes to your thinking?" 
Finally, if an examinee mentioned the proper criterion and also explained 
its relevance to the chosen answer but did not show how this relevance was 
derived from some general principle, the examinee was probed very 
indirectly so as to be given a chance to add more information. So, if a 
student mentioned that Ms. Vernon's being a driver education instructor 
and thus being more expert than Martine makes the difference, the 
examinee was probed: "So Ms. Vernon's being a driver education 
instructor makes the difference? - If an examinee merely responded "Yes", 
then he or she was asked to go on to the next question. If there was 
further explanation, this was recorded. 

The interview data was then used to rale the quality of examinees 1 
thinking. This information was coupled with the answers they had chosen 
to make judgements about whether or not particular i;ems and uie test as 
a whole had worked properly. Based on this feedback yet another version 
of the test was produced, Version C (See Appendix D), which was 
subjected to analysis through interview data collected in the same way as 
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that described above. When this stage had been reached the test seemed 
in satisfactory condition, except for a\ew minor cosmetic changes. These 
were made and the final version (See Appendix A) was administered to a 
large sample of students in order to acquirable reliability estimates and 
sufficient data for producing some noims. 

5.4. Summary 

This chapter has provided an overview of the developmental stages of 
the Test on Appraising Observations. The test has ev\lved through many 
stages, initially more through intuitive judgements of wnat seemed sound, 
later through data collected on small numbers of students, and finally 
through rigourously collected and .analyzed data on relatively large 
numbers of st ^ents. The aim was always the same: to produce a test for 
which good thinking generally led to good performance and poor thinking 
generally led to poor performance. The desire to have rather direct 
evidence on whether or not this aim was being met controlled more than 
anything else the methodology that was used. The systematic collection of 
think aloud protocols, their analysis, and the changing of items based on 
this analysis was a direct response to this aim. We believe that the Test on 
Appraising Observations meets this standard to a satisfactory degree. The 
degree to which it does is documented in the following chapter 
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Chapter 6 

« 

Data Collection and Analysis: 
Final Results 

Iu this chapter we describe in detail the data collection and analysis 
for the two experimental versions of the test, and show how the data was 
used in the transitions from Version B to Version C and from Version C to 
the final version. In addition, item and test statistics for the final version 
of the test are given. 

6.1. Samples and Data Collection 

Data on Version B was collected in the Fall of 1982 from two samples 
from seven senior high schools on the Avalon Peninsula in eastern 
Newfoundland. The communities ..varied from relatively isolated to 
relatively urban. The first sample was of 181 students in levels I and II (at 
the time Newfoundland did not have a three-year high school) and the 
second of 52 students fram the same grade levels. The first sample was 
chosen on the basis of intact classes. Each class was administered Version 
B of the test and asked to try to finish the entire instrument. Eight classes 
in all, four in each of levels I and II , were given the test in this way. The 
second sample consisted of students randomly chosen from classes in the 
same schools which had not been administered Version B, and randomly 
assigned to one of us for testing. Each student in this sample was asked to 
take either Part A or Part B of tfce test, and to take the first half of the 
p t assigned in the interview format and the second half in the normal 
testing format. Table 6-1 illustrates the sampling formal. 
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Table 8-1: Distribution of Samples by 

Grade Level, School, and Testing Format 



Group Testing Interview Testing 

School Level I Level II Level I Level II 



1 X X 

2 X X XX 

3 XX 

4 XX 

5 X X 

6 XX 

7 X X 



Data on Version C was collected in the Spring of 1983 from two 
samples of students from four schools in eastern Newfoundland. Two of 
the schools were in relatively urban settings while the others were in rural 
areas. The first sample which administered the entire test in intact classes 
and consisted of 171 students in levels I and II of the senior high school. 
The second group was interviewed individually on only one part of the test 
and consisted of 44 students. 

Data on the final version of the test was collected in the late Spring 
of from four senior high schools in southern Ontario. Students were 
administered the test in intact classes and in addition to the Test on 
Appraising Observations (TAO) were administered either the Cornell 
Critical Thinking Test, Level X (CCTT) or the Watson-G laser Critical 
Thinking Appraisal, Form A (W-G). The tests were administered through 
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the cooperation of Professor Philip Nagy who provided guidance to the 
schools involved. Each cooperating teacher received a copy of the 
instruction sheet in Appendix E. The testing was done through the 
schools' science departments in biology, chemistry, and physics classes. 
Consequently, the sample was biased towards students of average to above 
average achievement. Sampling was as shown in Table 6-2. 

Table 6-2: Number of Students in 

Ontario Samples for TAO, CCTT, and W-G 



School 


TAO 


W-G 




Bluevale 


101 


45 


42 


Forest Heights 


297 


129 


153 


Guelph 


100 


65 


36 


Waterloo 


106 


34 


42 



6.2. Basic Data and Derived Data 

6.2.1. Basic Data 

The basic data for this study came from two sources, students' 
choices of answers to questions on the tests and thinking protocols collected 
during interviewing. Answers to questions were recorded by students on 
standardized answer sheefl!»for each of the tests. The interviewed students 
typically verbalized their answers in addition to recording them on their 
answer sheets. If the verbalized answer differed from the recorded answer 
(which was the case only seldomly), then no remark was made by the 
interviewer. We wished answer selection errors to be present in our data 
since in normal testing situations they cannot be eliminated. Thinking 
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protocols were tape recorded and in addition we made brief notes while 
students were thinking aloud. These notes were afterwards checked for 
accuracy against the tape recordings. Finally, from the answer sheets and 
the think aloud protocols a number of variables were derived which served 
as the basis for evaluating the quality of the test. 

6.2.2. Derived Data 

Eight variables were derived from the basic data collected in the 
study: performance scores, thinking scores, item/test biserial correlation 
coefficients, item difficulty levels, item-thinking/item-performance biserial 
correlation coefficients, item thinking/performance index scores, 
respondent thinking/performance index scores, and Kuder-Richardson-20 
reliability indices. 

Performance scores . Performance scores were giveu as number of 
items answered correctly according to the accepted key. No correction was 
made for guessing. For various analyses it was advantageous to derive 
performance scores for particular parts of the test, either Part A or B, or 
for the part on which students were interviewed and the part completed 
before or after being interviewed. 

Thinking scores . The test was divided into four logical divisions 
according to the story line; Part A, items 1-15 and 16-28, and Part B, 
items 29-37 and 38-50. Students were interviewed on one of these sections, 
thus having to report their thinking on from 9 to 15 questions. Thinking 
scores were based upon an analysis and evaluation of these reports. For 
each item, thinking was rated according to the scale in Table 6-3. Table 
6-t shows how a student's thinking might be rated for item 3 of the 
test. For each item students could receive a thinking score of between 0 and 
3, and depending on tLe part of the test being taken could receive a total 
thinking score for that part of between 27 and 45. The effective maximum 
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thinking score per item was, as it turned out, 2. Respondents rarely 
thought so well as to receive 3 for an item. Thus, total thinking scores 
realistically had maximums of 18 and 30 depending on the part of the test. 
Table 6-3: Rating Scale for Thinking Scores 



Rating Basis of Evaluation 



Rating = 1 the respondent cites the criterion by which correct 

appraisal of the underlined statements may be made, or 
the respondent uses the criterion in comparing the two 
underlined statements but does not explicitly cite the 
criterion 

Rating = 2 the respondent cites the criterion and also uses it to 

compare the two underlined statements 

Rating = 3 the respondent cites the criterion, uses it to compare the 

two underlined statements, and also generalizes from the 
particular situation to situations like it 

Rating = 0 ihe respondent does none of the above or does noi 

respond 



Item/test biserial correlation coefficients . ' For each item, the biserial 
correlation between item performance (right or wrong) and total score on 
the test was calculated. This statistic was calculated only for those 
students who were in the non-interview groups and who thus completed all 
sections of the test. 

Item difficulty levels . Difficulty levels as measured by the proportion 
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Table 6-4: Rating Thinking for Item 3 



Rating Basis of Evaluation 



Rating = 1 The student points out that Mr. Wang was involved in 

the accident. 

Rating = 2 The student points out that Mr. Wang was involved in 

the accident, and compares Mr. Wang's involvement 
with Ms. Vernon's being a bystander. 

Rating = 3 The student points out that Mr. Wang was involved in 

the accident, compares this with Ms. Vernon's non- 
involvement, and shows how this is an ins;ance of a 
more general phenomenon. For instance, the respondent 
might say that people involved in situations where they 
might be blamed tend to be less believable than those 
who cannot be blamed. 



of students getting items correct were calculated for each item. Since this 
index was given as a proportion of respondents getting an item correct, the 
lower the index the harder the item was assumed to be for the group as a 
whole. Note the technical definition of difficulty in terms of overall group 
porformance. however, since it may not coincide with everyone's concept of 
difficulty. For example, there is no necessity to conceive item difficulty 
such that greater difficulty leads to poorer performance For example, one 
might imagine greater difficulty leading to greater motivation and thus to 
hotter or equal performance. Like the item/test biserial correlation 
coefficients this statistic was calculated only for those students who had 
completed the entire test. 

lt«'in-thinking/item-pt»rformnnce biserial correlation coefficients . The 

5tt 
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biserial correlation between item thinking scores and item performance 
scores was calculated for each item across subjects. This produced an 
index for each item of the relationship between thinking well on that item 
and getting the item right. One problem with the coefficient is that in the 
extreme (and in cases close to the extreme) when all (or most) subjects 
receive 0 for thinking as well as 0 for performance on an item, the biserial 
correlation yields an index of 0. However, in our opinion an item with a 
majority of (0,0) scores for (thinking, performance) is working well. When 
people think poorly they get the item wrong. Furthermore, in certain cases 
the biserial correlation coefficient is greater than unity, presumably 
because certain assumptions are violated by the data, normality being one 
of me most important assumptions for this statistic. To help capture the 
relationship between item thinking and item performance the index 
described in the following section was devised to be used in conjuction with 
the biserial correlation. 

Item-thinki ng/performance index scores . In order to try to offset 
somewhat the deficiencies of the item-thinking/item-performance biserial 
correlation, a thinking/performance index (T/P index) score was 
developed. First of all, combinations of thinking and performance scores 
were rated as in Table 6-5. Thinking scores from 0 to 2 only were chosen 
since there were so few scores of 3 for thinking. Any thinking score of 3 
was thus converted to 2. There were thus six possible combinations of 
thinking scores and performance scores, (T,P) scores. Combinations (0,0) 
and (2,1) were judged to give the same degree of positive evidence for the 
quality of an item (it being assumed as discussed in the chapter on test 
validation that one sign of quality items is that -ood thinking leads to 
••Direct responses and poor thinking to wrong responses), and were assigned 
a rating of +:>. Combinations (0,1) and (1,0) were judged to provide the 
Mime degree of negat.ve evidence for the quality of an item, and were 
assigned a rating of -1. (2.0) was judged to provide a higher degree of 
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negative evident than either of the previous two combinations and was 
assigned a rating of -2. Finally, the combination (1,1) was judged to 
provide an interm. d.ate level of positive evidence for item quality and was 
rated +1. 

<?* 

The aim was to provide an index of the average degree of evidence 
from different sources on the quality of an item, not to provide an index of 
cor-elation as with the biseTial coefficient. Thus, to arrive at the 
'hinking/performance index score for an item, the scores for all subjects 
answerfrig an item, as determined by Table 6-5, were averaged and the 
resuit divided by 2. The index thus ranges from -1 to +1, with -1 
indicating the highest level of negative evidence against the quality of an 
item, +1 the highest level of positive evidence, and 0 that the positive and 
negative evidence was in balance. 

Table 6-5: Weights of Evidence for 

Thinking/Performance Combinations j 



Thinking Scores 

0 1.2 

Performance 0 +2 -1 -2 

Scores 1 -1 +1 +2 



Respondent thinkin, performance index scores . Once 
thinking/performance evidence weightings per student and per item were 
rale i 'ed, there were several possibilities of derived scores. One was the 
item thinking/performance index score just discussed. Another is a 
thinking/performance index score for the respondent, calculated by 
avpraging evidence weightings obtained by that respondent for all items on 
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the test. This index gives a rating of the degree of evidence for the quality 
of the test as a whole as obtained from that one respondent. 

Kuder-Richardson-20 reliability indices . KR-20 reliability estimates, 
giving a lower bound on the reliability of the test, were calculated using as 
data the responses of those who completed ail items on the test. 

6.3. Analysis: Version B to Version C 

A 

The primary concern in all of our analyses was to determine the 
extent to which test performance could be explained by level of critical 
thinking, and adjustments were made in situations where there were 
systematic tendencies for some other factor or factors to explain 
performance. For this reason, considerable weight was placed on the 
thinking/perjormance index scores. Other indicators were considered but 
always in light of the thinking/performance relationship. In particular, the 
following questions were addressed and changes were made in light of the 
answers found: 

1. What was the relationship between thinking and performance 
for each item? 

2. What was the relationship between thinking and performance 
for the test as a whole? 

3. How does performance compare on items measuring the same 
principle? 

4. Flow is performance on each item related to overall test 
performance? 

5. Was "test.wiseness" a factor? 

B. Did the test systematically mislead in any way? 
7. Was reading difficulty a factor? 
S. Wore the instructions clear? 
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9. Were any of the test's characteristics systematically related to 

the ability of the students? 

i 

10. Did any of the factors, interviewing, interviewer, test section, 
grade level, or sex of examinee, affect thinking scores? 

11. Did any of the factors, interviewing, interviewer, test section, 
grade level, or sex of examinee, affect performance scores? 

The information contained in Table 6-6 as well as the student 
protocols served as the primary information for answering the above 
questions. 

8.3.1. Thinking/Performance Relationships for Items 

The relationship between thinking and performance for each item 
was indicated by both the item-thinking/item-performance biserial 
correlation coefficient and by the item thinking/performance index score. 
It is a matter of informed judgement, however, which magnitudes of these 
numbers should signal a warning. An obvious signal occurs if either of the 
numbers is negative. A negative biserial correlation would indicate that for 
that item poor thinking tends to lead to correct performance and good 
thinking to incorrect performance. A negative thinking/performance index 
score would indicate that there is more evidence against the quality of the 
item than there is in favour of it. 

Beyond this, however, it is difficult to know what should be taken as 
a low T/P index score or a low biserial correlation. A lower bound of 
acceptability for the T/P index can be derived on the assumption that a 
T/P index score should at least be as high as the score which would obtain 
for a sample of students randomly guessing their answers and all thinking 
poorly. In thi* situation, one-third of the students would get the item 
correct and all students would receive zero for thinking. Fop^. sample of N 
students. N'/3 would have a (T.P) combination of (0,1) yielding an evidence 
weight of -1 from Table 6-5, and 2N/3 would have a (T,P) combination of 
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Table 6-6: Item Level Statistics, Version B 

« 



Item 
No. 



Item/ 

Test 

biser. 



Diff. 
Level 



T/P 
biser. 



T/P 
index 



Item 

No. 



Item/ 

Test 

biser. 



Diff. 
Level 



T/P 
biser. 



T/P 
index 



1 

I 


.131 


o 
I 


tor 

.185 


'J 

•5 


.354 


4 


.301 


0 


.186 


/> 

0 


.296 




.182 


8 


.437 


9 


.150 


10 


.417 


11 


.074 


12 


.201 


13 


.282 


14 


.374 


15 


.123 


16 


.313 


17 


.409 


18 


.447 


19 


.411 


20 


.327 


21 


.417 


')•) 


.299 


23 


.419 


24 


.290 


25 


.488 



.702 


.622 


.757 


.936 


.425 


.787 


.287 


.689 


.514 


1.169 


.547 


1.063 


.249 


.206 


.558 


.568 


.276 


.867 


.751 


.623 


.398 


-0.727 


.354 


.717 


.238 


.451 


.331 


-0.162 


.215 


.863 


.348 


.654 


.628 


.552 


.556 


1.338 


.606 


.927 


.539 


.034 


.472 


.149 


.228 


.430 


.263 


.420 


.302 


.0 


.475 


.959 



.462 


26 


.731 


27 


.616 


28 


.615 


29 


.885 


30 


.769 


31 


.577 


32 


.385 


33 


.769 


34 


.346 


35 


-.231 


36 


.538 


37 


.458 


38 


.208 


39 


.625 


40 


.536 


41 


.715 


42 


.964 


43 


.786 


44 


.250 


45 


.250 


46 


.429 


47 


.429 


48 


.679 


49 


.769 


50 



.538 


.449 


.292 


.469 


.365 


.299 


.067 


.188 


.248 


.335 


.349 


.466 


.358 


.577 


.222 


.431 


.462 


.401 


.428 


.593 


.403 


.494 


.520 


.655 


.211 


.456 


.350 


.314 


.428 


.605 


.333 


,333 


.459 


.491 


.433 


.401 


.264 


.184 


.400 


.384 


.501 


.685 


.476 


.493 


.448 


.636 


.475 


.588 


.329 


.176 



.527 


.333 


.344 


.413 


.902 


.750 


.0 


.875 


.551 


.423 


.460 


.346 


.907 


.692 


.838 


.654 


.846 


.538 


.574 


.769 


.623 


.461 


.999 


.731 


1.296 


.334 


1.006 


.833 


.829 


.625 


-0.322 


.250 


.269 


.375 


.556 


.209 


1.006 


.87b 


.723 


.667 


1.240 


.958 


1.264 


.833 


1.192 


.917 


.848 


.708 


.886 


.750 



(0,0) with an evidence weighting of +2. Such an occurrence would give a 
T/P index score of .5. Note that this is a reasonably high score since from 
two-thirds of the sample the highest possible evidence weighting of +2 is 
obtained. Thus, choosing a T/P index of .5 as a lower bound of 
acceptability is quite conservative. 
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An examination of Table 6-6 shows that to one decimal place there 
were 16 items with T/P index scores less that .5 and one, item 11, had a 
negative score. Each of these items was treated as suspect and were 
examined in light of the other indicators of quality and in light of the 
protocols. In the end, all but two of the items, numbers 22 and 23, were 
altered. These items were close to the standard of acceptability in any 
case, and we thought that changes which were made to neighbouring items 
20 and 21 would make 22 and 23 stronger items as well. The strategy was 
to try not changing then, since no obvious way of changing thein was 
apparent, and to see how they behaved in the next trial. 

The changes that were made to items ranged from single word 
changes, to the addition qi deletion of material, to changes in the 
instructions of the test. As a result of the protocols and T/P index scores 
extensive changes were made to the introduction to the traffic accident 
*tory. In the Version B Introduction the names of all the characters and 
what they were doing at the time of the accident were provided. Since 
there were several characters in the story with different roles, we thought 
that providing the names' in a single list would assist students in keeping 
them straight. However, there were unexpected problems caused by doing 
this. For the first six items many students referred to the introduction for 
evidence to support their choice of answers. While thi? is a legitimate 
thinking strategy, it contributed to uncontrolled influences on students' 
responses and thence to unjustified interpretations of thinking from 
performances. For example, in item 1 the keyed answer is that Martine's 
statement that there were three cars at the scene of the accident is more 
believable. Good critical thinking would lead to this response because 
Martine who was driving would tend to be more alert to the number of 
( ars than Pierre who was reading a map and trying to figure out which 
way to go. However, eight students chose the correct answer by referring 
to the introduction and counting the number of cars mentioned there. One 
indent reasoned as follows: 
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(The first statement] is more believable because Martipe was in 
the car. too and well, when I read back there [pointing to the 
introduction], well, there were three cars, so she would have to 
be right. Maybe Pierre, "he said there was five cars. Maybe he 
v/as doing something else at the time of the accident. 

Now as it happens the introduction does not say how man/ cars were 
<at the intersection, but does mention three cars which were involved in the 
accident. Thus, a student who was not thinking critically would assume 
that the number of cars at the intersection equalled the number of cars in 
the accident, whereas the critical thinker would realize the fallacy oi this 
reasoning. Thus, the non-critical thinker would be rewarded with getting 
the item correct through an unsound reasoning process. 

The T/P index score for item 1 indicates that this item seemed to 
work satisfactorily despite the type of influencp just described. However, 
other items fared less well. For example, for items 11 and 14 (T/P indexes 
of -.231 and .208 respectively) some students still referred to the 
introduction for information which led them to choose the correct response 
but not for the desired reasons. This contributed to the poor behaviour of 
these items and provided more motivation for altering the introduction. 

It can be seen from the above examples that the computed 
relationships between thinking and performance were used in conjunction 
with the protocols of students' thinking. If the relationship between 
thinking and performance was low, a reason for this was often found in the 
protocols, which provided sound information on the types of corrective 
measures to take. This marks a substantial improvement over trying to 
make changes to items based on item/test biserial correlations and item 
difficulty levels without information on those factors which influenced 
students' responses. 
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6.3.2. Thin ting/ Performance Relationship for the Test as a 
Whole 

The thinking/performance relationship for each item is obviously 
related to this relationship for the test as a whole. However, the latter 
statistic is useful in its own right. Table 6-7 gives the correlation between 
thinking scores and performance scores for the entire test, and for both 
parts of the test separately. 

Table 6-7: Pearson Correlations Between Thinking 

Scores and Performance Scores 



Test Section Pearson's r r Significance 



Vnole Tes 


0 


.68 


0 


.46 


> 


.001 


Part A 


0. 


59 


0, 


.35 


> 


.001 


Part B 


0. 


77 


0, 


.59 


> 


.001 



The correlations obtained were judged satisfactory, at least at this 
stage in the test development. They were of the same order of magnitude 
as the KR-20 reliability which was computed to be 0.72. The relatively 
lower correlation obtained for Part A compared to Part B was explainable 
in terms of the considerable difficulty which was caused in Part A by 
examinees referring to the introduction for assistance. This was not a 
pioblem lor Part B. Thus, no changes were made to the test based on this 
information, but a caution was registered to check for an imbalance 
between the two parts in subsequent versions. 
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6.3.3. Items Testing the Same Principle 

As described previously, each item on the test was designed to test 
one of the principles in Table 3-1. While we were not overly concerned 
with internal consistency in the test as a whole, as measured by the KR-20, 
say, we judged that performances on at least those items measuring the 
same principle should correlate well. It is reasonable to argue that the 
ability to appraise observations is multi-dimensional and that the various 
aspects need not correlate well. However, this argument is not available 
for items testing the same principle. At the same time, it must be realized 
that even for items measuring the same principle, there was always a 
slightly different context. It is known that context affects reasoning 
substantially, 50 so we cannot expect the correlations to be perfect even 
were there no invalidating influences operating. 

Instead of computing correlations between items testing the same 
principles we chose to examine the item difficulty levels and the item/test 
biserial correlations. For the above reasons we did not demand that these 
be nearly equal nor did we automatically alter items just because they were 
substantially different. Instead, when the figures differed substantially (by 
about 50%) we took that as a reason to treat the item suspiciously. Table 
6-8 gives the item difficulties and the item/test biserial correlations for 
items designed to test the same principle. 

As noted in the table there are a total of 6 principles for which either 
the difficulty levels or the biserial correlations of their items differ by 50% 
or more. This involves 14 items in total, of which 10 were altered for the 
subsequent version. The remaining 4 items were unchanged because we 
felt that changes made elsewhere would remove the problems with these 
items. 



Kvans. J St. B.T. The psychology of deductive reasoning. London: Routledge k 
Kegan Paul, 1982. 
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Table 6-8: Item Difficulties and Item/Test 

Biserial Correlations for Items Testing 
the Same Principles 





iieni 


L/iiiicuiiy 


item/ 1 est 






Level 


Biserial 


I* 


22 


.228 


2QQ 




29 


.188 


067 




40 


605 


428 




43 


401 


433 


II. 1 


5 


301 


186 




33 


431 


222 


II.2* 


1 


702 


131 




35 


593 


428 


II.3 


3 


425 


354 




49 


588 


47^ 


II.8 


17 


628 


40Q 




47 


493 


476 


11.10 


26 


449 


538 




36 


494 


4fl3 


11.11* 


12 


354 


201 




15 


215 


123 


III 1 


25 


475 


488 




42 


491 


459 


III.2* 


23 


263 


419 




32 


577 


358 


III.3* 


39 


314 


350 




48 


633 


448 

• 110 


III.4.e 


20 


539 


327 




21 


472 


417 


IV.2 


27 


469 


292 




31 


466 


349 


IV. 4 


24 


302 


290 




45 


.384 


.400 


IV.8 


14 


.331 


.374 




50 


.176 


.329 


IV.9 * 


38 


.456 


.211 




41 


.333 


.333 


IV. 11* 


7 


.249 


.182 




11 


.392 


.074 


IV. 13 


9 


.276 


.150 




13 


.233 


.282 
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6.3.4. The Relationship of Item Performance to Overall Test 
Performance 

i 

It is typically assumed that performance on individual items should 
be substantially positively related to performance on the test as a whol* 
At the bottom of the assumption is the feeling that if all the items *\re 
designed to test "the same thing", then similar results should be achieved 
across items. Of course, if this "same thing" is a multifaceted thing, the 
assumption does not hold up. When the domain is multi-dimensional one 
can imagine situations in which performance on one aspect of the domain is 
not related at all to performance on some others, or even that the 
performances are negatively related. However, just because situations like 
this, though conceptually possible, are difficult to imagine we decided to 
flag any items whose item biserial correlation was essentially zero or was 
negative. 

Table 6-6 shows that there were no negative item/test biserials and 
only two (for items 11 and 29) which were essenti^y zero. Item 11 also 
had a troublesome thinking/performance relationship and was thus 
modified. Item 29 had good thinking/performance characteristics but a 
small change was made to make the first underlined statement less 
complex, and a note made to see how the item behaved in the subsequent 
version. 

6.3.6. Test Wiseness 

Test wiseness is ill-defined. In a rough and ready way we might say 
that test wiseness affects people's performance on a test if their previous 
experience taking tests has taught them how to do well on the test through 
means that are irrelevant to what the test is designed to measure. In 
addition, this tfea can be applied to the internal workings of one particular 
test if experience taking earlier sections of a test affect in irrelevant ways 
performance in subsequent parts of the test. 
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We checked for the effect of test wiseness by examining the protocols 
and by comparing average performances on different parts of the test. 
Through the protocols we found that some students chose answers on the 
basis of the sheer amount that was said by the characters in the story. For 
example, for item 30 a student chose the first underlined statement (which 
is the keyed answer) because it "explained more - , meaning that it said 
more. There has been advice in the testing literature for many years to be 
careful of this type of unwanted influence on test performance, so all 
questions of the test were examined so that (within particular items) no 
character in the story gave noticeably more information than another. 

We also checked for whether there was any systematic tendency for 
students to do better in one part of the test or another. Recall that each 
student took either Part A or Part B of the test and that each of these 
parts was in turn divided into two sections. Students were interviewed on 
one section and took the other section in the normal testing manner. 
Tables 6-9 and 6-10 show how performance scores and thinking scores 
compared for these groups. Table 6-9 must be read diagonally. For 
example, cells 1 and 6 contain the average performance scores of a group of 
students which was first interviewed on Part A, section I, and then took 
Part A, section II, in the normal testing format. Cells 7 and 4 by contrast 
contain the average performance scores of a group which first took Part B, 
section I. in the normal testing format and then was interviewed for Part 
B. section II. The la. a show that average performance scores are higher 
for the second s^. u : ^n taken regardless of the part of the test and 
regardless of whether a group was interviewed before answering questions 
under normal testing conditions or whether interviewed afterwards. The 
group represented by cells 7 and 4 is slightly anomalous, since their 
increase from section I to II is not as great as in the other groups. 

On the face of it this increase in performance from section I to II 
regardless of other factors might be taken as evidence that test wiseness is 
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influencing performance. However, there is support for the test wiseness 
hypothesis only when what brings about the increase in scores is irrelevant 
to what the test is designed to measure. However, Table 6-10 shows that 
in addition to an increase in average performance scores from section I to II 
there was an accompanying increase in average thinking scores. This 
suggests that whatever is the precise cause of the increase, it appears to be 
quite relevant to what the test is supposed to measure, that is, ability to 
use principles of thinking to appraise observations. It must be granted that 
students learned something from doing the test, that this was independent 
of what, if anythir?, they learned from the interviews, but that the 
learning was quite relevant to the purpose of the test. That is the 
improvement in performance, since accompanied by an $ improvement in 
thinking, was likely caused by legitimate reasons and not for something 
that might be called test wiseness. 

One qualification must be added. The average thinking scores in 
Table 6-1 J cannot be directly compared by group to the" performance 
scores in Table 6-9. Table 6-10 contains data on four groups. For 
example, the group interviewed on Part A, section I, is not the same as the 
group interviewed on Part A, section II. Thus, without assumptions of 
initial group equivalence the scores cannot be directly compared. Though 
the Ns are small, 13 per cell, students were randomly assigned to groups 
and average performance scores on section I (either Part A or Part B) were 
e>sentially the. same for all four groups. Thus, there is a strong 
presumption created of group similarity. The conclusions on test wiseness 
must, however, be tempered by the possibility of non-equivalence. 
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Table 6-0: Average Performance Scores by Test Section 

and by Order of Interviewing 



Part A p ar t fl 

Sect. I Sect. II Sect. I Sect. II 

Cell Number 1 

Interviewed 49 

Cell Number 5 

Not Interviewed 48 

Table 6-10: Average Thinking Scores by Test Section 



Part A Part B 

Section I Section II Section I Section II 

34 52 29 32 

6.3.6. Misleading Factors, Reading Difficulty, and Clarity of 
Instructions 

This section addresses a number of issues all of which have to do with 
whether students were able to comprehend the test. To check the overall 
reading level of th» test we used a method which offers suitable accuracy 
f«.r our purposes and incredible simplicity of application, Fry's Readability 
f ormula. 0 ( sing the m ,hod, the Test on Appraising Observations has a 

Fry. K A readability formula that saves time. Journal of Rinding 11 l%8 
•1 1. •'.-"» Ifi. ' ' 



2 3 4 

64 47 52 

6 7 8 

61 48 65 
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reading difficulty level of between sixth and seventh grade. So, even if 
Fry's method is considerably inaccurate the test should present no reading 
difficulties for most senior high school students for whom the test is 
intended. Despite this low measured reading level, we found during the 
analysis of interviews that there were vocabulary problems in twelve items, 
either because students did not know the meaning of particular words or 
because the meaning was ambiguous. Adjustments were made to correct 
for each of these difficulties. 

We have already discussed problems which arose with the 
introduction to Version B. However, the instructions also presented 
difficulties for some students. After reading the instructions in Version B 
some students were still confused about what to do. We had provided an 
example item in the instructions but we had shown students only how to 
mark their answers for the item, not how they might think through it. We 
thus elaborated the example, showing how someone thinking critically 
might think about the item and choo/e and mark an answer. 

6.3.7. Test Characteristics and Student Ability 

We were concerned that some of the test's characteristics might be 
systematically related to the critical thinking ability of students. In 
particular, we were concerned that the relationship between thinking scores 
and performance scores for the test might be dependent upon students' 
ability. Specifically, if the thinking/performance relationship was related 
systematically to performance on the test, then we would consider this a 
significant problem. 

Figure 6-1 is a scattered .' individuals' performances on the test 
motion they took against the biserial correlations of their performance 
s.-ores and their thinking scores. A vLual examination of the plot indicates 
that there is little systematic relationship, an impression confirmed bv a 
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calculated Pearson's r of 0.12, which is non-significant There is, then, no 
systematic relationship between the thinking/performance relationship and 
test performance. 

Figure 6-1: Scatterplot of Test Performance versus 
Thinking/Performance Biserial Correlation 
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Biserial Correlation 



Theoretically, in the ideal limit there should be a perfect relationship 
for all individuals between their thinking scores arid performance scores. 
This is the reason for expecting no relationship between performance scores 
and the thinking/performance relationship: the thinking/performance 
relationship should be the same, namely unity, regardless of performance. 
For this reason it is interesting to examine the distribution of 
thinking/performance relationships across individuals. The correlations 
range from very slightly negative to unity. Thus, there is substantial 
deviation from the theoretical limit of all correlations being unity. 
However, the median of the distribution is 0.70 indicating that a majority 
«»f the correlations are substantial. Given a KR-20 reliability of 0.72. this is 
about the most that could be expected. 
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8.3.8. Extraneous Influences on Performance Sc/res and Thinking 
Scores 

^rc ^. For any test it is important to know whether test performance is 

related to factors like the sex of the examinees and their grade levef. In 
addition, in the test development reported here it is important to know the 
effect, if any, of such things as being interviewed on the performance and 
thinking of the examinees. ' If the* interviewing affected people's 
performance or if the particular person conducting the interviews was a 
factor, then the value of the interviewing technique as a means of 
collecting validity information would be diminished. 

In order to test whether the interviewing had an effect on test 
performance an analysis of variance was performed with individuals' 
performance scores on the four test sections as the dependent variable, and 
testing format (interview or non-interview), interviewer, test section, grade 
level, and sex of examinees as the independent variables. Table's- 11 shows 
that none of the main effects or the two-way interactions were significant. 
In particular, the interviewing (format) seems not to have affected 
performance -scores, leading us to conclude that the information gathered in 
the interview 'was a trustworthy indicator 'of how the test would work in 
non-interview situations. 

Since performance scores and thinking scores were so highly 
correlated it was decided to perform a multivariate analysis of variance 
with thinking scores and performance scores as the dependent variables. 
The results of this analysis are contained in Table 6-12. T- v was a grade 
level effect significant at >.01, and a marginal test, sectr-b effect 
significant at about .10. These effects are tolerable and represent no 
invalidating information. In fact, the grade level effect might be expected. 
There were no significant effects found for the interviewer or for the sex of 
the examinees. This is a desired r«' -lit since effects from these sources 
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Table 8-11: ANOVA Results for Version B: 

Performance Scores by Testing Format, 
Interviewer, Test Section, 
Grade Level, and Sex of Examinee 



Source of Sum of Mean Significance 

Variation Squares DF Square F of F 



Fozma"* 1.25 1 1.25 .004 .960 

Test Section 3762 3 1254 2.52 .234 

Interviewer 63.9 1 63.9 .200 .660 

Grade Level 3245 1 3245 5.19 .263 

Sex 355 1 355 1.72 .415 

For X Int 309 1 309 .947 .334 

Int X Ta 1491 3 497 1.52 .215 

Int X Gl 626 1 628 1.92 .170 

Int X Sex 207 1 207 .632 .429 

For X Ts 1811 3 604 1.55 .145 

For X Gl 65.4 1 65.4 .200 .656 

For X Sex 543 1 543 1.66 .201 

Tb X Gl 385 3 128 .393 .758 

Ts X Sex 1015 3 338 1.04 .382 

Gl X Sex 182 1 182 .559 .457 

Within 16410 46 357 

would indicate either that our interview information was untrustworthy or 
that the test was being influenced by what should be an irrelevant factor. 
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Table 6-12: MANOVA Rosults for Version B: 

Performance Scores and Thinking Scores byj 
Interviewer, Test Section, Grade Level, J 
and Sex of Examinee */ 



Hypotheeie Error 
Spurce of Dependent Mean Mean Significance 
Variation Variable Square Square (Wilks Lamda) 



Grade Performance 2555 
Level Thinking 195 



191 



154 



003 



Test 



Performance .598 



Section Thinking 



577 



191 



154 



.11 



Inter- Performance 131 



viewer Thinking 167 



191 



154 



.549 



Sex of Performance 332 
Examinee Thinking 271 



191 



154 



.317 



6.3.9. Summary 

This concludes the analysis of iho da' a collected on Version B of the 
test. The aim in all the ana'y.?es was to determine the extent to which test 
performance could be explained by level of critic*; thinking, and to what 
extent unwanted factors influenced performance. Our conclusion was that 
the overall structure of the test was suitable, but that m*ny changes would 
have to be made at the i?»m a;; J instructions level. We hoped that these 
local changes would not affect the over; *' behaviour of the instrument, as 
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described in the ANOVA and MANOVA results, and in the analysis of test 
wiseness, reading, clarity, etc. 

It has been stressed before that test design is an iterative procedure 
in which one constructs a prototype and adjusts it until it behaves suitably. 
The designer is never certain, however, that the upcoming version will be 
the final one. Thus, we proceeded to construct Version C (see Appendix D) 
according to the changes indicated thus far and then to collect data similar 
to that collected for Version B. 

0.4. Analysis: Version C 

The following questions most concerned us in evaluating the 
performance of Version C: 

1. Did the thinking/performance relationships for items improve 
over Version B? 

2. Did the thinking/performance relationship for the test as a 
whole improve over Version B? 

3. Was performance on i'ems measuring the same principles more 
uniform than in Version B? 

I. Were performance scores as unaffected by unwanted influences 
as in Version B? 

6.4.1. Thinning/Performance Relationships for Items 

On Version B there were 16 items with T/P index scores less than .5 
and one of these with a negative index score. All of these items but two, 
items 22 and 23, were altered. We thought that changes made to 
neighbouring items would favourably affect these two items. Table 6-13 
mmparrs T/P index scores per item for Version B and Version C. 

Note first that there are no negative T/P index scores for \ <>rsion 
('. In addition, the average index score for Versiop C is .656, an increase 
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Table 6-13: Comparison of T/P Iifuex Scores Per Item 

for Version B and Version C 



Item 


Version 

▼ v. i O I v AA 


Version It^m 


Version 


v ersion 




B 






B 




c 


1 


.462 




Jo 


333 




.667 


2 


731 


.ooO 


11 


.413 




.<08 


3 


.61G 


J 1 0 


Jo 


.750 




.i 08 


4 


• .615 


.040 


on 


.875 




.850 


.) 


.885 


. i ol 


on 
oU 


.423 




.800 


6 


.769 


.DiO 




.346 




.800 


** 
i 


.577 


.OUo 


00 

oJ 


.692 




.700 


8 


.385 


.OOO 


OO 


.654 




.300 


9 


.769 


.000 


o4 


.538 




.i 50 


10 


.346 


.4JO 


0£ 
OO 


.769 




.500 


11 


-.231 


.040 


Oft 
OO 


.461 




.450 


12 


.583 


.Oof 


0/ 


.<31 




.800 


13 


.458 


.004 


OO 
OO 


.334 




4 r\r\ 

.400 


11 


.208 


.4 Jo 


on 
o9 


n n o 

.833 




r r a 

.550 


5 


.625 


.692 


40 


.625 




r r a 

.550 


16 


.536 


.625 


41 


.250 




.650 


17 


.715 


833 

• OOO 


4 A 


.O/0 




.250 


IS 


.964 


.958 


43 


.209 




.oou 


19 


.786 


.917 


44 


' .875 




.700 


20 


.250 


n c o 

.yoo 


45 


.667 




.500 


21 


'>50 

• M *J \J 


.917 


46 


.958 




.750 




.429 


.750 


47 


.833 




.950 


OO 


4 on 


.833 


48 


.917 




.800 


21 


.679 


.500 


49 


.708 




.750 


2", 


T69 


.958 


50 


.750 




.950 










Mean= 


=.58 


Mean=.66 










SD=.219 


SD=.205 




over the average of , 


,578 for V< 


?rsion B. Of the 15 items 


on Version B with 




positive but < 5 ind 


ex scores, 


10 had their T/P index raised to 


over .5 


on 




Version C. Only one 


item of th 


e remaining five received 


a lower 


T/P ind 


ex 
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on Version ('. In addition, however, 6 items on Version C obtained index 

r 

scores <.5 which had received scores >.5 on Version B. We turned to the 
protocols collected for Version C and to the patterns of thinking and 
performance scores to see if any weaknesses in these items could be 
discovered. 

For items 3, 4, 7 and 8 we discovered a common error in students 
thinking which was largely responsible for the low T/P index scores and 
which seemed easy to fix. In each case there were two or more students 
who made their choice on the basis of one of the characters in the story 
being a driver of an automobile and the other not being a driver. For 
example, in item 3 th ree students incorrectly chose the first statement as 
more believable because Mr. Wang was a driver and would be looking at 
the traffic closer than a bystander. The obvious fix was to make both 
characters in each of these four items either a driver or a non-driver, 
effecting the type of control which is sought in test items. 

There remained seven items, numbers 10, 11, 12, 14, 33, 38, and 42, 
which had T/P index scores of <.5. For each of these items most of the 
negative evidence leading to the low T/P index scores came from a (0,1) 
combination of thinking/performance scores, that is, from examinees who 
had answered the item correctly but had received a zero for thinking. For 
each of these items the (0,1) (T,P) combinations were matched by an equal 
or greater number of (0,0) combinations. This is a phenomenon which we 
think is unavoidable by the best test. When examinees do not understand 
the nature of the problem facing them (as indicated by a 0 thinking score) 
but they are faced with a multiple-choice format, then their performance 
scores will often appear as random guesses. That is, for every item that an 
examinee gets incorrect because his or her thinking was poor/the person 
will get a proportionate number correct despite thinking poorly. From 
examining the protocols this appeared to be the mechanism operating for 
these seven items. The students who tho* ;ht poorly but got the items 
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correct appeared simply to be lucky in choosing the correct answer. We 
were not able to detect anything systematic occurring which led students 
to pick the correct answers despite poor thinking. That is, the low T/P 
index scores seemed to reflect more a characteristic of the students with 
respect to the subject matter in these particular items than it seemed to 
reflect a characteristic of the items. 

6.4.2. Thinking/Performance Relationship for the Test as a 
Whole 

The computed Pearson correlation between thinking scores and 
performance scores was 0.75. This amounts to an r 2 of .56 and is 
significant at >.001. The corresponding correlation for Version B was 
smaller, being 0.68. This improvement was judged to be important, and 
along with the data presented in the previous section, illustrated that the 
protocols collected on the Version B trial were quite informative in leading 
to changes which enhanced the validity of the the instrument. In addition, 
this level of correlation is of the same order as the computed KR-20 
reliability coefficient of 0.74. 

6.4.3. Items Testing the Same Principle 

fi 

It was observed for Version B that item difficulty levels and item/test 
biserial correlations were dissimilar for some pairs of items measuring the 
same principle. This occurred for 6 principles and involved 14 items. We 
argued that a certain degree of difference (up to about 50%) is tolerable 
because even though items might be testing the same principle, the 
differences in their surrounding context could give rise to legitimate 
variation. In the end, 10 items were altered to correct for this difficulty. 

The changes in the 10 items appropriately affected the characteristics 
of items testing three of the principles, II. 2. 11.11. and IV. 11. However, for 
the remaining three principles. I. III.2, and III.3. difficulty levels or 
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item/test biserials or both continued to differ by more than 50% from each 
other. In addition, the data for Version C indicated more that S0% 
differences for three other principles, II. 1, III.l, and IV.2. Thus, for 
Version C there remained 6 principles for which our 50% rule of thumb 
was violated. On examining each of the items involved, they were found to 
have suitable characteristics in other regards. For example, the T/P index 
scores for the items were quite high. Without having any unambiguous 
information on how to improve the items, we decided to let the items 
remain as they were. We suspect that there are contextual influences 
operating in these questions, but without considerable research into the 
influence of context on people's critical thinking ability we can do little 
more than speculate, and to produce the best items we can with the 
knowledge that is currently available. 

6.4.4. Extraneous Influences on Performance Scores and Thinking 
Scores 

As for V ersion B, ANOVA and MANOVA analyses were performed 
in order to check for unwanted influences on performance and thinking 
scores. The ANOVA results as shown in Table 6-14 with performance 
scores as the dependent variable, and testing format (interview or non- 
interview), interviewer, test section, grade level, and sex of examinees as 
the independent variables. The analysis showed significant effects for Test 
Section and Format by Sex interaction. The Test Section effect was of 
little concern, since there is no specific reason for wanting the difficulty of 
both sections of 'the test to be the same. The Format by Sex interaction 
was, however, of more concern. It did not cast doubt on the validity of the 
test, necessarily, bu: it did call into question the applicability of the 
interview technique in obtaining evidence for validity. However, there was 
nothing to recommend by the finding in terms of changes to the test, and 
<$i\en that interactions between Sex and other variables were not 
significant, and °;iven that our main concern that there be no main effect 
for Sex was satisfied, we merely noted the finding. 
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Table 6-14: ANOVA Results for Version C: 

Performance Scores by Testing Format, 
Interviewer, Test Section, 
Grade Level, and Sex of Examinee 



Source of 
Variation 


Sum of 
Squares 


DF 


Mean 
Square 


F 


Significance 
of F 


Format 


18.0 


1 


18.0 


1.62 


/ .424 

/ 


Test Section 


3615 


3 


1205 


6.84 


/' 

/ . 074 

/ 


Grade Level 


741 


1 


741 


1.95/ 


/ 

.395 


Sex 


41.8 


1 


41.8 




.754 


Interviewer 


31.3 


1 


31.3 


.127 


.722 


For X Int 


11.1 


1 


11.1 


.045 


.832 


Int X Te 


528 


3 


176 


.717 


.545 


Int X Gl 


379 


1 


379 


1.55 


.219 


Int X Sex 


254 


1 


254 


1.03 


.313 


For X Te 


83.7 


3 


279 


.114 


.952 


For X Gl 


180 


1 


180 


.732 


.395 


For X Sex 


1086 


1 


1086 


4.42 


.040 


Ts X Gl 


1511 


3 


504 


2.05 


.116 


Tb X Sex 


190 


3 


63.4 


.258 


.855 


Gl X Sex 


33.2 


1 


33.2 


.135 


.714 


Within 


9320 


42 


222 
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The MANOVA was performed with thinking scores and performance 
scores as the dependent variables and Grade Level, Test Section, 
Interviewer, and Sex of Examinees as the independent variables. As with 
Version B, there was a Grade Level effect, though marginal at about the 
.11 level. In addition, there was a Test Section effect as with Version B, 
but with, Version C the effect was more highly significant. As hoped, there 
were no significant Interviewer or Sex of Examinee effects. Results are 
contained in Table 6-15. 

Table 6-15: MANOVA Results for Version C: 

Performance Scores and Thinking Scores by 
Interviewer, Test Section, Grade Level, 
and Sex of Examinee 



Hypothesis Error 
Source of Dependent Mean Mean Significance 
Variation Variable Square Square (Wilks Lamda) 



Grade Performance 430 126 

Level Thinking .282 150 

Test Performance 14C4 126 

Section Thinking 862 150 

Inter- Performance 7.92 126 

viewer. Thinking 351 150 

Sex of Performance 15.9 126 

Examinee Thinking 7.03 150 



.108 



.005 



165 



843 



79 

Version C seemed to be a better test than Version B, given the 
improved T/P index scores and the equ ./ good resistance to extraneous 
influences. However, it appeared that making improvements from version 
to " ers i on } la( j f inal i y reacne d a point of diminishing returns. While 
Vcrvon C vvas better than Version B, it seemed to us that improvements to 
Versi- i C would not be worth the expense of new data collection. In 
addition, the data collected on Version C, unlike with previous versions of 
the test, did not often point unambiguously to changes that should be 
made. As just discussed, there were some indications that items could be 
improved, but little information on what sort of change would lead to 
improvement. Therefore, instead of just "shooting in the dark" we 
decided to settle for a few cosmetic changes. 

To us it seems that we have driven this test construction about as far 
as it can go without a substantial increase in knowledge abot. . nat 
influences test performances and, in particular, about hov ritical thinking 
is influenced. For example, while we have argued that any good critical 
thinking test must be set in a context because sound appraisals cannot be 
made devoid of context, there is little knowledge on the effect of context 
on critical thinking. 52 Thus, given the state of knowledge, we were fa.ed 
with an unsolvaMe problem when items purporting to test the same critical 
thinking principle, but doing so in different contexts, displayed different 
statistical characteristics. Was this due to differential effects of context on 
( ritical thinking performance? If so, it would not point to a probl- rn with 
the Ipse. Was the difference due to items of varying quality? If so, this 
would reduce 'he validity of the test. Until more is knoun. these quest urns 
< ;in ^ot be answered. 



■ "t 

"\'<>rns, S P The choice of standard conditions in defining critical iSink...g 
••<imp< tpnee. Educational Theory, .15. forthcoming. 
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0.5. Final Data 

Data on the fiial version of the test (see Appendix A) was collected 
from four high schools in southern Ontario as indicated in Table 6-2. In 
addition, data was collected on the Cornell Critical Thinking Test, Level 
X and the Walson-G laser Critical Thinking Appraisal, Form A, which 
were used to compare the characteristics of the Test on Appraising 
Observations. Performance scores only were calculated (the key to correct 
answers for ^e Test on Appraising Observations is given in Appendix F) 
as each test was given in normal testing circumstances. An item analysis 
was performed which yielded item difficulty leve's, item/whole-test biserial 
correlations, and KR-20 reliability estimates The results for all four 
schools combined is given in Table 6-16, and for the schools taken 
separately in Tables 6-17 to 6-20. 

Table 6-16: Item Analysis Results for All 

Southern Ontario Schools Combined 



it 



Mean 



Variance 



KR-20 



Test on Appraising 31 

Observations (50)* 

Cornell Critical 51 

Thinking Test (76)* 



Watson-Glaser Critical 49 
Thinking Appraisal (80)* 



32 



41 



78 



.69 

.72 

(.77-. 81)** 
.80 

(.69- 85)** 



* Total number of items on test 

** Range of KR-20 reported in test manual 

*** Range of split-half reliabilities reportud in 

manual 
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Table 8-17: Item Analysis Results for 

Bluevale School 



Test 



Mean 



Variance 



KR-20 



Test on Appraising 30 
Observations 

Cornell Critical 50 
Thinking Test 

Watson-Glaser Critical 44 
Thinking Appraisal 



41 



42 



165 



Table 8-18: Iterr Analysis Results for 
Forest Heights School 



76 



74 



j2 



Test 



Test on Appraising 
Observations 

Cornell Critical 
Thinking Tes« 

Watson-Glaser Critical 
Thinking Appraisal 



Mean Variance KR-20 



31 



51 



49 



30 



3d 



54 



.68 



71 



72 



In terms of average reliability estimates the Test on Appraising 
Observations is the lowest of the three. This is no doubt in part due to the 
fact that it is the shortest of the three te?*s, containing 50 items compared 
to 73 an< . tcr the other two tests. The difference also appears partly 
due to two cases of extreme data in the samples. The variance on the Test 
on Appraising Observations for the Waterloo sample was lower than any 
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Table 8-19: Item Analysis Results for 

Guelph School 



Teat Mean Variance KR-20 



Test on Appraising 31 35 .73 

Observations 

Cornell Critical 50 49 .78 

Thinking Test 

Watson-Glaser Critical 51 59 .76 

Thinking Appraisal 



Table 8-20: Item Analysis Results for 

Waterloo School 



Test Mean Variance KR-20 



Test on Appraising 32 23 .58 

Observations 

Cornell Critical 51 37 .71 

Thinking Test 

*r,tsca-Glasar Crttica-1 50 52 .74 

thinking Appraisal 

• •tiKT measured variance, which contributed to the lowest reliability 
1." for that test. By contrast the variance on the Watson-Claser 
('ntir<-' flunking Appraisal for the Bluevale sample was by far tie 
highest variance recorded, contributing to the high measured reliability, if 
thesr two -xtremes are eliminated from consideration then the spiead in 
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reliability among the three tests would diminish Overall, we considered 
the reliability calculated on the entire sample, .69, while low in terms of 
some psychological instruments, to be adequate given the reliability of the 
other critical thinking tests. 

In addition in terms of some other test characteristics the Test on 
Appraising Observations fared better than the other instruments Near 
the end of both the Cornell Test and the Watson-Glaser Test, there was a 
tendency for items to have zero or negative item/test biserial correlations. 
This occurred for 7 items on both tests, but did not occur on our test at 
all. This suggests, but only suggests, that some new factor begins to 
operate near the end of the other two critical thinking tfc.„. This might be 
fatigue, due to the length of the test, but this »s purely speculation. It 
would be an interesting issue to pursue, however. 
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Chapter 7 
Summary and Conclusions 

This report has described the design of a test of one aspect of critical 
thinking ability, the ability to correctly appraise observations. The test is 
intended primarily for an audience of senior high school students, though it 
might be used effectively at other levels, particularly with undergraduate 
students. Although the test is intended to be a power test, most senior 
high school students finish it in one class period, not allowing for 
administrative time to pass out materials and instruct students in how to 
take the test and mark their answers. The test, can be scored easily by 
hand, but no machine scoring system is yet in place. 

The Test on Appraising Observations is based upon a comprehensive 
set of principles for appraising observations and upon a particular theory of 
test validation. The principles- were described and presented in Chapter 3. 
Although they are subject to modification, the principles as they currently 
stand offer the most comprehensive and defensible set knowi to us. The 
theory of test validation was presented in Chapter 4 and differs from 
nirr »t|y accepted theory in its emphasis on the discovery of the mental 
mechanisms which lead people to perform on tests the way they do. The 
theory has given rise to a validat ; on methodology focussed on extracting 
fr<»tn examinees their thinking while they work through questions on the 
test. Chapter 6 describes how this information was subsequently used in 
assessing the validity of the test. ' 

In the version in Appendix A the Test on Appraising Observation* 
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represents an instrument which has evolved through several stages with 
attendant improvements at each stage transition. The test rates 
favouraoly on standard measures of qilality with two the the most^wlcfely 




used critical thinking tests, the Cornell Critical Thinking Test, Level X 
and the Wa ts on -G laser Critical Thinking Appraisal, Form A. In addition, 
this test has been subjected to intensive scrutiny through the protocols of 
about 100 students. We know that for the most part when students 
perform well on this test they do So because they have thought well, and 
when they do poorly they have thought poorly. According to the view of 
test validation outlined earlier this knowledge is at the foundation of any 
daim to a vest's being valid. We know there are and always will be 
exceptions. However, with care the instrument can be effectively used. In 
addition, because the test concentrates on only one aspect of critical 
thinking ability, it provide* a better indication of people's ability in that 
area than the currently available tests which are designed to give an 
assessment of critical thinking ability in general. 

We have also discovered in this study that protocols of examinees' 
thinking can be efi -otively collected and used in designing a test. The 
protocols were used to identify problem questions, but because we had 
detailed information on the cause of people's performances we were usually 
able to turn to the protocols for specific changes that had to be made for 
items to behave more adequately The information was very accurate in 
many < a>es. Thus, we hope that this report illustrates the effective 
ernpio'inent of a validation technique which has a long record <>f 
endorsement but only a sparse record of use. 
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Appendix A 

Test on Appraising 
Observations 
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This test tells you two stories. Read them very carefully. 
As you read the stories you will be asked to answer 
questions about what people say. You must read ALL the 
information you are given. EACH piece of information may 
be needed to answer some questions. Each question has 
only ONE accepted answer. 



DO NOT WRITE ON THIS BOOKLET. 
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DIRECTIONS 

J° f a f h ; u ff t,on v ° u wi " be A iven two statements in bold type. You must choose which 
statement in bold type, if either, you have MORE reason to believe at the time the statements are 
made. 

Remember: Choose- between the statements in bold type only. You may use statements 
which are not in bold type to help you choose. 

Here is an example: 



0. Two friends. Cathy and Helen, are driving along a country road. Suddenly an animal runs in 
front of the car and crosses to the other side of the road. 

Cathy says, look! There is a small brown animal!" 

Helen says, "Cathy, you are wearing dark-coloured sunglasses. That animal, was grey." 

To answer this question, first look for some important difference between the people or the 
situations. In this case, Cathy is wearing sunglasses. Cathy s sunglasses could have made the 
animal appear a different colour. Since Helen criticizes Cathy, it seems that she is not wearing 
sunglasses. Therefore, Helen would have a better view than Cathy. People who have a better 
view of things tend to be more believable. 

Since you have MORE reason to believe the SECOND statement in bold type, Helen s, at the 
time the statements are made, you should mark your answer sheet like this: 



First Second Neither 

0 0 • « o 

Mark your ans-ver sheet now for question 0. 

In the rest of the test questions, mark your answers as follows: 



First 


Second 


Neither 


• 


o 


0 


0 


• 


O 


0 


0 


• 



Means \ >u have more reason to believe the FIRST 
statement at the time the statements are made. \ 

Means you have more reason to believe the SECOND 
statement at the time the statements are made. 

Means you have no more reason to believe EITHER 
'tatement at the time the statements are made. 



When answering a question, do NOT use information given in liter questions. You MAY use 
information given in earlier questions. Fcr example, suppose you are working on question 10 
You MAY use information in questions 1 to 9. You may NOT use information in questions 1 1 to 

rM J 
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PART A 

A Traffic Accident 

A traffic accident has just occurred at an intersection which has a stop sign in each direction. 
Several cars were involved. 

A policeman and a policewoman will question people. Later several investigators will collect 
information about the accident. It isjpeur job to judge the evidence given in the statements that 
follow. 1 

1 . A policeman is questioning Pierre and Marline. They were in their car at the intersection but 
were not involved in the accident. Martine is the driver and Pierre, who had been trying to 
figure out which way to go, is the map reader. 

The policeman asks Martine hovymany cars were at the intersection when the accident 
occurred. She answers, "There wV© three cars." 

Pierre sayc, "No, there were five cars." 

a 

2. A small boy and his father had been standing on the sidewalk when the accident occurred. 
The boy says, "There was a motorcycle at the intersection." 

His father says, "No, there was no motorcycle at the intersection." 

3. A policewoman has been asking Mr. Wang and Ms. Vernon questions. She asks Mr. Wang, 
who was one of the people involved in the accident, whether he had used his signal. 

Mr. Wang answers, "Yes, I did use my signal." 

Ms. Verpon had been driving a car which was not involved in the accident. She tells the 
officer, "Mr. Wang did not use his signal. But this didn't cause the accident." 

4. The policewoman then points to Ms. Rosen's car which was one of the cars involved in the 
accident. She asks whether Ms. Rosen had signalled. 

Mr Dawe, another driver not involved in the/accident, says, "Ms. Rosen signalled. I was just 
talking to Ms. Vernon about this and I'm sire she will agree with what I said." 

Martine says. "Ms. Rosen did not signal. I'm sure I'm right." 

5 The policeman talks to Mr. and Mrs. Peters, who were also involved in the accident. It is easy 
to see that Mr. Peters, who was the driver, is very upset by the accident. The policeman asks 
him to estimate his speed just before the accident. 

Mr. Peters says, "I was going about 15 kilometers an hour." 

A little later when he is feeling well he says, "I was going about 30 kilometers an hour." 



The policeman asks whether or not theTeters' car had stopped at the stop sign. Ms. Vernon, 
who is a driver educatmninstructor, says, "I am very experienced in these matters. The 
Peters' car did not stop. 

Martine. who overheard this conversation, goes up to the officer and says, "The Peters' car 
did stop at the stop sign." 
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. The officer turns to question Martine and Pierre and Mr. Dawe. The officer asks them to 
estimate the speed of Mr. Wang's car when it hit the others. 

Mr. Dawe says, "It was going about 40 or 45 kilometers an hour. " 

The officer says, "It was going faster than that, wasn't it?" Martine says, "Oh yes, it was 
going about 60 or 65 kilometers an hour." 



8. Martine adds, "Mr. Wang want right through the stop sign." 

The police officer turns to Mr. Dawe and says that at the scene of the accident Mr. Dawe 
couldn't remember whether Mr. Wang had stopped at the stop sign or not. Mr. Dawe says "I 
remember now, Mr. Wang did stop at the stop sign." 

9. Ms. Vernon then says, "I also remember that a fancy blue sports car went through the atop 
sign." 

Martine says, "A car with twin headlights went right through the stop sign." 

10. Martine says, 'Three cars collided at the same time. There was one crash." 

Ms. Vernon says, "There was more than one crash. It would be very strange for the three to 

collide at exactly the same time." 

1 1 The police officers ask the people involved in the accident and the other drivers to come to the 
police station to make official statements. At the station, the policeman questions Mr. Peters. 
Mr. Peters points to a drawing of the intersection and says, "Just before the accident 
occurred Mr. Wang's car approached the intersection from that direction." 
The police officer says to Mr. Peters, "Surely Mr. Wang's car came from a different 
direction." "Oh yes," says Mr. Peters, "it did come from a different direction." 

12. The policeman turns to Mr. Dawe to question him. In the background they can hear a 
conversation between the other officer and some of the other witnesses. Some are 
discussing whether one of the cars went through a s op sign. 

Mr Dawe says, "Mr. Wang and Ms. Rosen crashed into each other. I saw it happen '» 
' Also, I remember that a car went straight through a stop sign, too." 

13 Nearby, the policewoman and Martine are looking at the drawing of the intersection. 

Martine says, "A short time before the accident everyone was driving normally." 

She continues, 'Then there was a loud squeal of tires. Mr. Peters' car turned quickly 
toward the fruit stand on the corner." 

1 4 The policewoman asks Mr. Dawe to tell in which direction Mr. Peters was travelling before 
the accident Mr. Dawe says. "He was going toward Fifth Street." 

The policewoman looks at her notes which were made at the scene of the accident. At that 
time Ms Vernon had pointed and said that Mr. Peters was going away from Fifth Street 
before the accident. 
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15. The policewoman remarks that many people turn left at this intersection even though they 
are not allowed. She says that this causes many accidents. She asks Martine to continue to 
tell what she remembers. 

Martine says, "Ms. Rosen came to a complete stop.'' 
She then adds. "But then she turned left." 

1 6. Meanwhile, at the scene of the accident several inspectors have been collecting information 
about the accident. They are examining the wrecks and the marks on the road. Two teams 
were collecting information separately. They are now finished and are comparing notes. 

Inspector Suzuki says. "Our notes say that Ms. Rosen's car skidded 427 centimeters 
before hitting the other cars. I made the measurement and also made the notes." 

Inspector Rousseau says, "According to our notes Ms. Rosen's car skidded 457 
centimeters before hitting the other cars, nspector O'Reilly measured the skid by herself 
and Inspector Smith copied down what she said." 

17. Inspector Rousseau says. "We also measured the length of Mr. Wang's skid. We used a 
measuring tape that was 1000 centimeters long. Inspector O'Reilly held one end at trie 
begi nning of the skid and I took the reading at the other end. I wrote down the measurement. 
Mr. Wang's car skidded 320 centimeters." 

Inspector Rossi Sv'ys. "I also measured Mr. Wang's skid. I used a 30 centimeter measuring 
stick. I started by placing one end at the beginning of the skid and by putting a mark at the 
other end. I then placed the beginning of the stick at that mark, and so on until I reached the 
end of the skid. I wrote down my measurement. Mr. Wang's car skidded 350 centimeters." 

18. Inspector Rousseau says, "I found some brown paint on the left front fender of Mr. Wang's 
car. I looked at it with a magnifying glass. It is the same colour as the paint on Ms. Rosen's 

car." 

Inspector Ross* says, "I also studied that paint on the left fender of Mr. Wang's car. I looked at 
it under the microscope. It is not the same colour as the paint on Ms. Rosen's car." 

1 9 Inspector Smith, who does not use a microscope often, says, "I'd like to check that myself." 
He looks at the paint sample under the microscope and says, "There are no gold-coloured 
spots in this sample." 

Inspector O'Reilly, who uses a microscope often, looks at the sample. "There are gold- 
coloured spots in the sample/' she says. 



20. Inspector Rousseau and Inspector Smith have been using cameras which develop pictures 
instantly to take pictures of the accident. Inspector Smith's camera is an older model and is 
more difficult to adjust. They compare pictures of the skid marks of Ms. Rosen's and Mr. 
Wang's cars. They are trying to find out who stopped faster. 

Inspector Smith points to his pictures and says. "Mr. Wang's skid marks are darker than 
Ms. Rosen's.'' 

Inspector Rousseau looks at his pictures and says, "No, Mr. Wang's skid marks are no 
darker than Ms. Rosen's skid marks." 
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21. Both Inspector O'Reilly and Inspector Rousseau have taken pictures of the cars involved in 
the accident. Inspector O'Reilly says, "My pictures show that Ms. Rosen's and Mr. Wang's 
cars were damaged the same amount. I took several pictures of each car by itself after they 
were rolled apart." 

Inspector Roussea u says, "My pictures show that Ms. Fosen's car was damaged more than 
Mr. Wang s car. I took several pictures of the pile-up before the cars were rolled apart." 

22. Inspector O'Reilly says. "Mr. and Mrs. Peters' car is only slightly damaged " 
She continues, "The accident probably wasn't Mr. Peters' fault." 

23. Inspector Rossi and Inspector Suzuki examine the pictures taken by O'Reilly and Rousseau 
vVang C 's°cfi S r U ' Uki 9130068 * * PiCtUfe a ° d 8aV8, " Jh * r ° * a Part han9ln9 down under Mr ' 

Inspector Rossi studies the picture for several seconds and says, "That's not part of Mr 
Wang's car. That's a shadow." 

24. They then turn to examine the wrecked cars. Inspector Rossi points and says, "Look the 
brakeline to the front brakes of Ms. Rosen's car is broken." 

Inspector Rousseau overhears this and says. "That's strange. I discovered about an hour ago 
that that brakeline was not broken. " 

25. Inspector Smith slides under Ms. Rosen's car to examine the brakeline. "The handbrake 
cable is broken," he says. 

Inspector Suzuki kneels down and peers under Ms. Rosen's car. "No" she says "the 
handbrake cable is not broken." 

26. Inspector Suzuki examines the brakeline of Ms. Rosen's car. She says, "This rubber hose in 
the brakeline is worn through. It must have happened gradually." 

Inspector O'Reilly, who thinks that Inspector Suzuki is always wrong, also examines the 
brakeline. "No." she says, "the rubber hose is cut. It must have snapped suddenly." 

27. Inspector Rossi checks the brake fluid container of Ms. Rosen's car. He tells the other 
inspectors that there is a small amount of fluid left. 

Inspector Smith chec* the fluid container as well and says. "There is no fluid left there." 
Inspector Suzuki checks as well and says. "There is a little left at the bottom." 

28 Inspector 0 Reilly says. "One of the police officers checked the brakes. He told me that he 
pressed the brakes and they worked. Ms. Rosen had at least partial braking power at the 
time of the accident." 

Inspector Rousseau says. "I just checked the brakes myself. I pressed the brakes and the 
pedal went straight to the floor. Ms. Rosen had no braking power at the time of the 

accident." 
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The investigators were eventually able to agree on all aspects of the investigation. They 
turned their report over to the insurance company. 



THIS IS THE END OF PART A. 

IN PART B A NEW STORY BEGINS. 

THE INSTRUCTIONS ARE THE SAME AS FOR PART A. 

BEGIN PART 6 NOW. 



s • •- '. ■. 

PART B 

Exploring a River 

Imagine that you are a member of a group which is exploring one of your country's rivers. 
There are several people in the group who have^no advanced scientific knowledge. There are also v 
two scientists in the group. Professor Plant a biologist, is an expert in the study of living things: 
plants, insects, animals, and so on. Professor Rocks, a geologist, is an expert in the study of the 
Earth: the structure of rocks and minerals and the shape of its surface. Their names should help 
you to remember the subjects in which each is an expert. 

You have just arrived at a camping pladeijy the side of the river. It is early morning ana; just 
becoming bright. People in your group begin to explore the area around the camp. 

In what follows the rest of your trip will be described. You will be given statements rfiade by 
people in your group during your trip. As in Part A, you will be asked to judge how believable these 
statements are. 



29. You and some members of your group are getting a fire ready to cook breakfast. Others are 
looking at some mountains which are several miles away. Juanita,says, "Those mountains 
have several white streaks going from the top to the bottom/' 

Professor Rocks says to her, "Those streaks are small streams, I would say.'' 

30 Cheng says, "The white streak farthest to the right seems to divide into two parts halfway 
down the mountains." 

Scott says, "It does divide into two parts at that point. Some of it goes in one direction, the' 
rest goes in another." 

Professor Plant says, "It does not seem to divide into two parts." 
Ginette says, "I think Cheng is right. It does seem to divide into two parts." 
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32. Meanwhile. Mary, Juanita, Scott, and Cheng are walking through the campsite. Scott points 
to his right and says, "Look, there are two Swallows." 

Mary, who had been looking to Scott's left, turns quickly in the direction Scott pointed. She 
gets a quick glimpse of the birds and says, "Those birds are not Swallows. They are 
Chickadees." , 

33. Juanita says. "I was looking in the same direction as Scott. I saw the birds, too. Tfiey were 
Sparrows." 

Scott becomes upset at what Juanita says. He shouts, "I know what I saw.The birds were 
Swallows." 

34. Scott quickly calms down. Juanita jokes and reminds him that he often confuses birds. For 
example, yesterday he pointed to some Sparrows and said they were Juncos. Everyone had 
agreed he was wrong. Scott laughs and agrees that this was so. 

Juanita then says, "The birds to which Scott pointed this time were Sparrows. I had a 

good look at them." 

Scott responds, "I'm sure this time they were Swallows. I got a good look at them, too." 

35. Scott and Cheng leave Mary and Juanita. They join Professor Rocks and lie down near the 
edge* of a large grove of trees. Cheng says he is sleepy and closes his eyes. Professor Rocks 
and Scott are looking about. About one half hour passes. Cheng has not made a sound the 
whole time. His eyes are still closed. 

Suddenly Scott says to Cheng, "I hear a sound in those trees behind us." 
Cheng says, "There was no sound in those trees. Stop talking." 

36. Scott begins to read a book called Wild Animals of North America. It contains thingshe never 
heard before. He becomes fascinated by it. Professor Rocks is still looking arourid. 

"I hear another sound." Scott says. "It is a blackbear." 

"I heard a sound, too,'' says p rofessor Rocks. "But it was not the sound of a blackbear." 

t 

37. Professor Rocks continues, "According to Professor Plant, a team of scientists studied the 
wildlife of this area last summer. In their report they listed all the animals they sighted. They 
reported that they saw no blackbears." 

"Several of my friends visited this area last summer," replies Scott. "One told me in a letter 
that they saw some blackbears." 

) 

38. Scott and Professor Rocks agree to check with Scott's friend about seeing bears. They hear 
no more sounds. You finish your breakfast and start your hike downstream. In a little while 
Scott points to a small stream flowing into the main one. This stream is not pleasant looking. 
Its water is coloured orange. > 

Cheng is a member of an anti -pollution group. He sees the coloured water and becomes very 
angry He exclaims, "Some people are very careless! There is not a plant living in that 
water." 

Mary is a member of the same anti-pollution group as Cheng. She looks at the water too and 
says, "There are some plants living in the water." 
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39. Cheng looks at the water again and says, There are no insects in the water." 

He says, "There is a stream close by my house which I see nearly every day. It is dirty like this 
one. There are no insects in that stream, either." 

40. You continue down the river. Before long Scott yells, "I smell smokel I wonder if it's a forest 
. fire?" 

Everyone stops and sniffs the air. Cheng says, "Yes, it is a forest fire." 

41. You and your group decide to discover the source of the smell. You find a path leading 
through the forest in the direction from which it is coming. Cheng rushes ahead of the others. 

Cheng screams, "I see smoke up aheadl The forest is on firel" 

Ginette catches up with him, looks and says, "No, Cheng, that's not smoke. It is fog." 

42. As you walk along the path, you reach an area where you hear echoes of your voices from all 
directions. 

Mary says, "I hear a knocking sound straight ahead." 
Professdr Rocks says, "I see someone straight ahead." 

43. Your group rounds the next bend and Juanita says, "Look, there are some campers. They 
have a campfire burning." / 

Ginette says, "I would say the smoke we were smelling came from there." 

You continue down the path until you react' a log cabin. The owner is outside working on the 
woodpile. He greets your group and invites you all to stay for lunch. You accept. 

t 

44. While you are resting after lunch, Professor Rocks notices that Juanita, Ginette, and Scott 
are interested in the rocks of the area. She reaches into her knapsack and takes out two old 
notebooks. She says that the notebooks contain the records of two different scientific teams. 
Both teams were studying the rocks of the area when they made the records. Juanita, 
Ginette and Scott start to look through the records and to read some of the reports. One is a 
report of Lookout Mountain. 

Juanita says, "At the beginning of the record I am reading the writer says thatthe lake below 
Lookout Mountain is 1 154 centimeters deep. They found the depth by lowering a string 
with a lead weight tied to it until it reached the bottom. Then they measured the wet part of 
the string." 

Ginette says, "This record reports that the lake below Lookout Mountain is about 1 100 
centimeters deep. They describe their measuring method. It is the same as the one 
described in the record from which Juanita just read." 

45 Juanita says, "The report I am nowreading says that the fHflowing was recorded the day after 
the observatipy/i was made. According to the record there is a hot spring behind Lookout 
Mountain It reports that the hot spring shoots out water every thirteen minutes." 

Ginette says, "The report I am reading says that the following was recorded five minutes 
after the observation was made. It also talks about the hot spring behind Lookout Mountain. ~ 
However, it says that it shoots qut water every nine minutes." 
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46. Ginette says, This record says that tht water temperature ofthe hot spring is 1 1° C." 

Professor Rocks remarks, "That is strange. A hot spring should have a much higher water 
temperature -than that." 

Juanita says, "This record says that the temperature is 60° C." 

47. Scott says, "This record . sports the temperature of the water in a small stream near the hot 
spring. The report says that five separate readings were taken and the average of the five 
readings recorded. It reports that the temperature of the water is 25° C." 

"This record also reports the temperature of a small stream near the hot spring. It does not 
say whether more than one reading was taken. It reports that the temperature of the water 
is 20° C. Professor Rocks says. 

48. At this point everyone is well rested. You begin to follow a path which leads around a nearby 
lake. Suddenly an animal crosses your path. 

Juanita says, "Look, there's a small red squirrel." 

Ginette says, "I see it, tool" She runs after ita little and says, "I see it*againl That animal is 
not a red squirrel." 

V* 

49. Cheng, who is very fond of Juanita and often does things to try to make her think highly of 
him says, "There's another animal over there. It is a red squirrel like you were hoping to see, 
Juanita." 

"No," says Scott, "That animal is a shrew." 

50. You conti nue along the path. Cheng recalls. "Remember, Scott, we pitched our tent a short 
way from here last summer. We could see that mountain to the right of our tents." 

When you reach the camping spot, Scott says, "No, Cheng. Ourtent was here, We could see 
the mountain to our left, there. " 

I n another hour you come to a road. You have reached the end of your trip. A person is waiting 
to drive your group home as you had planned. 

THIS IS THE END OF THE TEST. 

CHECK YOUR ANSWERS IF YOU HAVE TIME. 
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Test on,? Assessing the Bel1evab1l1ty of Observation Statements 

VERSION B 



by Stephen Norrls and Ruth King 
Institute for Educational Research and Development 
Memorial University of Newfoundland 

1982- 



This test tells you two stories. As yo.u read the stories 
you will be asked to answer questions about what people say. You 
must read ALL the information you are given. EACH piece of 
information -nay be needed to answer some questions. Each question 
has only ONE accepted answer. To answer a question do NOT 
use information given 1n later questions. You MAY use 
information given in earlier questions. 
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Part A 



A Traffic Accident 



A traffic accident has just occurred at an intersection which has a 
stop sign in each direction. Cars driven by Mr. Peters , Ms. Rosen , and Mr. Wang 
were all involved in the accident. Mrs. Peters was riding with her husband. 

Ms. Vernon and Mr. Dawe had been standing on opposite sides of the 
street at the time of the accident. A, small boy of about six years old was 
standing next to Mr. Dawe. Marti ine and Pierre , two senior high- school students, 
had stopped at one of the stoo signs. They were not involved in the accident. 

A policeman and a policewoman will question these people. Later 
several investigators will collect Information about the accident. It is your 
job to judge the evidence given in the statements that follow. 

Here are all the people involved: 
People in the Accident People not in the Accident 

Mr. and Mrs. Peters in their car Martine and Pierre in their car 

Ms. Rosen in her car Ms. Vernon standing on the sidewalk 

Mr. Wang in his car Mr. Dawe standing on the sidewalk 

The small boy next to Mr. Dawe 

The Investigators 

Inspector Rousseau Inspector Suzuki 

Inspector O'Reilly Inspector Rossi 

Inspector Smith 

A Policeman and A Policewoman 
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"Part A (Instructions) 



In each question you will be given two underlined statements . You must 
choose which underlined statement , 1f e1ther/ "you have WORE reason Eb~~be11eve 
at the time the statements are made . 

Remember: Choose between the underlined statements only . You ma> u- 
statements which are not underlined to help you choose. 

Here 1s an example: 

A policeman 1s questioning Pierre and Marti ne. Marti ne 1s the driver and Pierre., 
who had been trying to flqure out which way to go, 1s the map reader. 

1. The policeman asks Marti ne how many cars were at the Intersection 
when the accident occurred. She answers, " There were three 
cars . 

Pierre says, "No, there were five cars ." 
A 

If you think you have MORE reason to believe the FIRST underlined 
statement, Marti ne's, at the time the statements are made, mark your answer sheet 
like this: 

First Second Neither 
1. • 0 0 I 

If you think you have MORE reason to believe the SECOND underlined 
statement, Pierre's, at the time the statements are made, mark your answer sheet like 
this: 

First Second Neither 
.1.0 • 0 

If you think you have NO MORE reason to believe EITHER underlined 
statement at the time the statements are made, mark your answer sheet ljke this: 

First Second Neither ( 

1. 0 0 • 

For number 1 mark the answer you think 1s correct. 




STOP. Wait for the signal to begin question 2. ( 
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Remember: Mark your answers as follows: 

FIRST: You have more reason to believe the FIRST statement at the time 
the statements are made. 

SECOND: You have more reason to believe the SECOND statement at the 
time the statements are toade. 

NEITHER: You have no more reason to believe EITHER statement at the time 
the statements are made. 



2. The small boy, who had been standing next to Mr. Dawe, says, " There was a 
motorcycle at -the intersection ." 

Mr. Dawe says, "No, there was no motorcycle at the intersection ." 

A policewoman has been asking Mr. Wang and Ms. Vernon questions. She asks Mr. 
Wang, who was one of the people involved in the accident, whether he had stopped 
at the stop sign. 

3. Mr. Wang answers, "Yes, I came to a full stop at the stop sign ." 

Ms. Vernon, who had watched the accident happen, tells the officer, " Mr. 
Wang's car did not come to a full stop ." 

The policewoman then points to Ms. Rosen's car whixh was one of the cars involved 
in the accident. She asks whether Ms. Rosen had signalled. 

4. Mr. Dawe says, " Ms. Rosen signalled . ; was just talking to Ms. Vernon and I'm 
sure she will agree with what I said." 

Martine says, " Ms. Rosen did not signal . I'm sure I'm right." 

The policeman talks to Mr. and Mrs. Peters, who were also involved in the 
accident. It is easy to see that Mr. Peters, who was the driver, is very upset by 
the. accident. The policeman asks him to estimate his speed just before the 
accident. 

5. Mr. Peters says, " I was going about 15 kilometers an hour ." 

A little later when he is feeling better he says, " I was going about 3 0 
kilometers an hour ." 

6. The policeman asks whether or not the Peters' car had stopped at the stop 
sign. Ms. Vernon, who is a driver education instructor, says, "I am very 
experienced in these matters. The Peters' car did not stop ." 

Martine, who overheard this conversation, goes up to the officer and says, 
" The Peters' car did stop at the stop sign . " 

The officer turns to question Martine and Pierre and Mr. Dawe. The officer asks 
them to estimate the speed of Mr. Wang's car when it hit the others. 

. Mr. Dawe says, " It was going about 40 or 45 kilometers an hour ." 

The officer says, "It was going faster than that, wasn't it?" Martine says,- 
"Yes, it was going about 60 or 65 kilometers an hour. " 
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8. Marti ne adds, " Mr. W^ftg-went right through the stop sign ," 

Mr. Dawe says, 'l^can't remember whether Mr. Wang' stopped at the stop sign or 
not. I think he did, though." A while later when the officer asks him again 
he says, " Mr. Wang did stop at ^he step sign ." 

9. Ms. Vernon then says, " I also remember that a blue car went through the stop 
sign ." 

Martine says, " A car with twin headlights went right through the stop sign ." 

10. Mr. Dawe says, "The three cars collided at the same time. There was one 
crash." 



- Ms. Vernon says, "No, the Peters' car hit an instant or so later. There was 
more than one crash . It would be very strange for the three to collide at 
exactly the same time . " 

The police officers ask the people involved in the accident to accompany them 
to the police station to make official statements. At the station, the policeman 
questions Mr. Dawe. 

11. Mr. Dawe says, " Just before the accident occurred Mr. Wang's, Ms. Rosen's, and 
Mr. Peter's carsTpproached the intersection ." 

The police officer asks, "Didn't you see any other cars, Mr. Dawe?;' "Oh yes," 
says Mr. Dawe, " there, was another car ." y 

In the background there has been a conversation between the other officer and some 
of the other fitnesses. Some are discussing whether one car went through a stop 
sign. '" « 

12. Mr. Dawe heard this and continued his testimony, " Mr. Wang and Ms. Rosen 
crashed into each other . I saw it happen." 

"Also, I remember that a car went Straight through a stop sign , too." 
Nearby, the policewoman is questioning Martine. 

13. Martine says, " Just before the accident everyone was driving normally ." 

She continuf "Then there was a loud squeal of tires. Mr. Peters' car £i/rned 
quickly toward the fruit stand ." 

14. The policewoman asks Martine to tell in which direction Mr. Peters was 
travelling before the accident. Martine says, " He was going in the direction 
of^the barber shop . " 

Tfie policewoman looks at her notes and sees that at the scene of the accident 
Ms. Vernon had pointed and said that mV. Peters was going away from the barber 
shop before the accident . 

The policewoman remarks to Martine that they have murh trouble trying to get people 
to use their directional signals. She said that they gave over one hundred tickets 
at one intersection a few days before. She asks Martine to continue to tell what 
she remembers. 
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15. Marti ne says* "Ms. Rosen came to a complete stop. 



She then adds, "But she did not use her signal ." ■ 

Meanwhile, at the scene of the 'accident several inspectors have been collecting 
information about the accident. They are examining the wrecks and the marks on the 
road. Two teams were collecting information separately. They are now finished and 
are comparing notes. 

16. Inspector Suzuki says, "Our notes say that Ms. Rosen's car skidded 427 
centimeters before hlttjng the other cars . / I made the measurement and al s o 
made the notes." ' 

Inspector Rousseau says, "According to our notes Ms. Rosen's car skidded 457 
centimeters before hitting, the other cars . Inspector O'Reilly measured the. 
skid and Inspector Smith copied down what she said." 

17. Inspector Rousseau says, "We also measured the length of Mr. Wang's skid. We 
used a measuring tape that was 1000 centimeters long. Inspector O'Reilly held 
one end at the beginning of , the skid and I' toek the .reading at the other end. 

' I wrote down the measurement. Mr. Wang's car skidded 320 centimeters ." 

Inspector Rossi says, "I also measured Mr. Wang's skid. I used a 30 
centimeter measuring stick. I started by placing one end at t^he beginning of 
the.skia and by putting a mark at the other end. I then placed the beginning 
of the stick at that mark, and so on until I reached the end of the skid. I 
wrote down my measurement. Mr. Wang's car skidded 350 centimeters ." 

18. Inspector Rousseau says, I found some brown paint on the left front fender of 
Mr. Wang's car. I looked at it with a magnifying glass. It Is the same r> 
colour as the paint on Ms. Rosen's car .' " 

Inspector Rossi says, "I also studied that paint on the left front fender of 
Mr. Wang's car. I looked at it under the microscope. It is not the same 
colour as the paint on Ms. Rosen's car ." 

19. Inspector Smith, who uses a microscope from time to time, says, "I'd like to 
check that myself." He looks at the paint sample under the microscope and 
says, " There are no gold spots in this sample . The paint on Ms. Rosen's car 
has gold spots." 

Inspector O'Reilly, who uses a microscope regularly, looks at the sample. 
" There are gold spots in the sample ," she says. 



Inspector Rousseau and Inspector Smith have been using cameras which develop 
pictures instantly to take pictures of the accident. Inspector Smith's camera is 
an older model and is more difficult to adjust. 

20. They compare pictures of the skid marks of Ms. Rosen's and Mr. Wang's cars. 
Inspector Smith points to his pictures and says, " Mr. Wang's skid marks are • 
darker than Ms. Rosen's . Mr. Wang must have been going faster than Ms. 
Rosen. " 

Inspector Rousseau looks at his pictures aid says, "No, Mr. Wang's skid marks 
are no darker than Ms. Rosen's skid marks . You can't tell f from my pictures 
who was going faster." 
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21. Both Inspector O'Reilly and Inspector Rousseau have taken pictures of the cars 
Involved in the accident. Inspector O'Reilly says, "My pictures show that Ms. 
RoseVs and Mr. Wang's cars were damaged the same amo unt. I took several $ 
pictures of each car after they were separated." - 

Inspector Rousseau ss 's, "My pictures show that Ms. Rosen's car was damaged 
more than Mr. Wang's car . I took several pictures of the pUe-up." 

22. Inspector O'Reilly says, " Mr. and Mrs. Peters' car is only si ightly 
damaged . 

She continues, " The Occident probably wasn't Mr. Peters' fault ." 

» 

23. Inspector Rossi and Inspector Suzuki examine the pictures taken by O'Reilly 
and Rousseau. Inspector Suzuki glances at a picture and says, " There is a 
part hanging down under Mr. Wang's car . " 

Inspector Rossi studies the picture for several seconds and says, " That's not 
part Qf Mr. Wang's car . That's a shadow." 

They then turn to examine the wrecked cars. 

24. Inspector Rossi points and says, "Look, the brake line to the front brakes of 
Ms. Rosen's car is broken ." 

Inspector Rousseau overhears this and says, "No, it is not broken . I 
discovered about one-half hour ago that the brakelines were not Drpken." 

25. Inspector Smith slides under Ms. Rosen's car to examine>the brakeline. The 
handbrake cabfle is broken ," he says. 

Inspector SuzuH kneels down and peers under Ms. Rosen's car. "No," she says, 
" the handbrake cable is not broken . " 

26. Inspector Suzuki Examines the brakeline of Ms. Rosen's car. She says, " This 
rubber hose in the brakeline is frayed . It must have worn through 

gradual ly . " 

Inspector O'Reilly, who always thinks that InspectdV Suzuki is wrong, also 
examines the brakeline. "No," she says, " the rubber hose is cut . It must 
have snapped suddenly." 

Inspector Rossi checks the brake fluid container of Ms. Rosen's car. He tells the 
other inspectors that there is a small amount of fluid left. 

27. Inspector Smith checks the fluid container as well and says, " There is no 
fluid left there >" 

Inspector Suzuki looks as well and says, " There 1s a little left at the 
bottom." 
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28. Inspector O'Reilly says, "One of the police officers checked the brakes. He 
t t old/foe that he pressed the brakes and they worked . Ms. Rosen had at least 
^Twftlal braking power at the time of the accident." 

Inspector Rousseau says, "I just checked the brakes myself. I pressed the 
bfrakes and the pedal went straight to the floor . Ms. Rosen had no braking 
power at the time of the accident." ~ 

The Investigators- were eventually able'to agree on- all aspects of the 
Investigation. They turned their ^report over to the Insurance company. 



STOP HERE. 

HO NOT GO ON UNTIL YOU ARE TOLD. IF YOU HAVE TIME, CHECK YOUR ANSWERS TO THIS 
PART OF THE TEST. 
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PART B. 



J 



Exploring A River 



Imagine that you are a member of a group which is exploring one of your 
country's rivers. There are several people in the group who have no advanced 
scientific knowledge. There are also two scientists in the group. Professor Plant 
a biologist, is an expert in the study of living things: plants, insects, animals, 
and so on. Professor Rocks , a geologist, is an expert in the study of the Earth: 
the structure of rocks and minerals and the shape of its surface. Their names 
should help you to remember the subjects in which each is an expert. 

You have just arrived at a camping place by the side of the river. It 
is early morningand just becoming bright. People in your group begin to explore 
the area around the camp. 

In what follows the rest of your trip will be described. You will be 
given statements made by people in your group during your trip. As in Part A, you 
will be asked to judge how believable these statements are. 

Remember: 

If you think you have MORE reason to believe the FIRST underlined 

statement at the time the statements are made, mark your answer sheet like 
this: 

First Second Neither 

• 0 t 0 

If you think you have MORE reason to believe the SECOND underlined 
statement at the time the statements are made, mark your answer sheet like 
this: 0 

First Second Neither 

Of 0 

If you think you have NO MORE reason to believe EITHER underlined 
statement at the- time the statements are made, mark your answer sheet like , 
this: 

First Second Neither 

0 0 • 



STOP. Wait for the signal to' beg in this part. 
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29. While you and some members of your group are getting & fire ready to cook 
breakfast Juanita says, " Those mountains over there, which are a few miles 
away, have several white breaks going from the top to the bottom? " 

Pre "essor Rocks says to her, " Those streaks are small streams , I would say." 

30. After looking at the mountains a bit longer Cheng says, " The white streak 
farthest to the right seems to split into two parts halfway down the 
mountains ." 

\ Scott says, " It does split into two parts at that point ." 

31. Professor Plant says, " It does not seem to spilt into two parts ." 

Ginette says, "I think Cheng is right. It does seem to split into two 
parts ." 

32. Meanwhile, Mary, Juanita, and Cheng are walking through the campsite. Scott 
suddenly points to his right and says, "Look, there are two Swallows ." 

• Mary, who had been looking to Scott's left, turns quickly in the direction 
Scott pointed. She gtts a quick glimpse of the birds and says, "Those birds 
are not Swallows. They are Chickadees ." 

33. Juanita says, "Mary was right, S&ott. The birds were Chickadees ." 

Scott becomes upset at what Juanita says. He shouts, "I know what I saw. 
The birds were Swallo ws . " 

Scott quickly calms down. Juanita jokes and reminds him that he often confuses 
birds. For example, yesterday he pointed to some Sparrows and said they were 
Juncos. Everyone had agreed he was wrong. Scott laughs and agrees that this was 
so. 

34. Juanita then says, " The birds to which Scott pointed this time were 
Chickadees . I had a good look at them." 

Scott responds, " I'm sure this time they were Swallows . I got a good look at 
them, too." \ 

Scott and Cheng leave Mary and Juanita. They join Professor Rocks and lie down 
near the edge of a large grove of trees. Cheng says he is sleepy and closes his 
eyes. Scott becomes fascinated by a b^ok called Wild Animals of North America . 
Professor Rocks is watching the sky. About one half hour passes. Cheng has not 
made a sound the whole time. His eyes are still closed. Scott is looking at his 
book. 

35. Suddenly Scott says to Cheng, " I hear a sound in those trees behind us." 
Cheng says, " There was no sound in those t^ees . Stop talking." 

36. "There was a sound," Scott insists. " It was a blackbear ." 

"I heard a sound, too," says Professor Rocks. "But It was not the sound of a 
blackbear." 
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37. Professor Rocks continues, "According to Professor Plant, a team of 
scientists studied the wildlife of this area last summer. In their report 
they nsted all the animals they sighted. They reported that they saw no 
blackbears . " 

"Two of my friends visited this area last summer," replies Scott. "He told 
me in a letter that they saw some blackbears ." 

Scott and Professor Rocks agree to check with Scott's friend about seeing bears. 
They hear no more sounds. You finish your breakfast and start your hike 
downstream. In a little while Scott points to a small stream flowing into the 
main one. This stream is not as pleasant looking as others you have passed. 

' ^ 

38. Cheng is a member of a group which is fighting to stop pollution of rivers. 
He sees the coloured water and becomes very angry. He exclaims, "Some people 
are very careless! There is not a living plant in that water ." 

"No, Mary says, There are some plants living in the water ." 

39. Cheng looks at the water again and says, " There are no insects in the water , 
though." 

He says, "There is a stream close by my house which I see nearly every day. 
It is dirty like this one. There are no insects in that stream , either." 

40. You continue down the river. Before long Scott yells, " I smell smoke ! I 
wonder if it's a forest fire?" 

Everyone stops and sniffs the air. Cheng says, 'Yes, it is a forest fire ." 

You and your group decide to discover where the smell is coming from. You find a 
path leading through the forest in the direction from which it is coming. Cheng 
rushes ahead of the others. 

41. Cheng screams, " I see smoke up ahead !' The forest is on fire!" 

Ginette catches up with him, looks and says, "No, Cheng, that's not smoke . 
It's fog rising from a pool of water there." 

As you talk, you hear echoes of your voices from all directions. ' 

42. Mary says, " I hear a knocking sound ahead ." 

Professor Rocks says, " I see someone ahead who appears to be 'swinging 
something ." 

43. Your group rounds the next bend and Juanita says, "Look, there are some 
campers. They have a campfire burning ." 

Ginette 'says, " The smoke we were smelling came from there ." 

You continue down the path until you reach a log cabin. The owner is outside 
working on the woodpile. He greets your group and invites you all to stay for 
lunch. You accept. 
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While you are resting after lunch, Professor Rocks notices that Juanita, Ginette, 
and Scott are interested in the rocks of the area. She reaches into her knapsack 
and takes out two old notebooks. She says that the notebooks contain the records 
of two different scientific teams. Both teams were studying the rocks of the area 
when they made the records. Juanita, Ginette and Scott start to- look through the 
records and to read some of the reports. One is a report of Lookout Mountain. 

44. Juanita says, "At the beginning of the record I am reading the writer says 
that there is a lake 1002.6 centimeters deep below Lookout Mountain . He says 
his team found the depth by lowering a string with a lead weight tied to it 
into the water until it reached the bottom. Then they measured the wet part 
of the string." 

Ginette says, "This record reports that the lake is about 1000 centimeters 
deep below Lookout Mountain . They describe their measuring method. It is 
the same as the one described in the record from which Juanita just read." 

45. Juanita. says , "The record I am now reading says that the following report was 
made the day after the observation was made. According to the record there 
is a geyser behind Lookout Mountain. It reports that the geyser shoots out 
water every thirteen minutes . " 

Ginette says", "This record says that each report was made within five minutes 
after the observation was made. It also talks about the geyser behind 
Lookout Mountain. However, it says that it shoots out water every nine 
minutes ." 

46. Ginette says, "This record says that the water temperature of the geyser is 
U°C . " Professor Rocks remarks, "That 1s strange. A geyser should have a 
much higher water temperature than that." 

Juanita says, "This record says that the temperature is 60°C ." 

47. Scott says, "This record reports the temperature of the water in a spring 
near the geyser. The report says that five separate readings were taken and 
the average of the five readings recorded. It reports that the temperature 
of the water is 55°C ." 

"This record also reports the temperature of a spring near the geyser. It 
does not say whether more than one reading was taken. It reports that the 
temperature of the water is 50°C ," Professor Rocks says. 

At this point everyone is well ; ted. You begin to follow a path which leads 
around the lake. Suddenly an animal crosses your path. 

48. Juanita says, "Look, there's a small brown animal . ^ 

Ginette says, "I see it, too! Isn't it grey? She runs after it a little 
and says, "I see it againl That animal is grey, not brown ." 

49. Cheng, who is very fond of Juanita and often does things to try to make her 
think highly of him, says, "I got a good look at the animal. It was brown , 
1 i ke Juanita said. " 

"No," says Scott, "the animal was grey. I got a good look at it, too." 
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v ou continue along the path. Cheng and Scott had camped in this area before. 

50. Cheng recalls, "Remember, Scott, we pitched our tent a short way from here. 
We could see that mountain tothe right of our tents . " 

When you reach the spot, Scott says, "No, Cheng. Our tent was here. We 
could see the mountains to our left , there." 

In another hour you come to a road. You have reached the end of your trip. A 
person is waiting to drive your group home as you had planned. 

t 

THIS IS THE END OF THE TEST. 
• CHECK YOUR ANSWERS TO THIS PART IF YOUR HAVE TIME. 
DO NOT GO BACK TO PART A. 



/ 
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Test on Assessing the Bel levabl 1 Ity of Observation Statements, Version B 



Answer Sheet SCHOOL 

GRADE ~ T 
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CTOCT 

r IR5T 


crrnun 

SECOND 


Mr t Turn 

NEITHER 




FIRST 


SECOND 


NEITHER 


1. 


0 


0 


0 


26. 


0 


0 


0 


2. 


0 


0 


0 


27. 


0 


0 


0 


3. 


0 


0 


0 


28. 


0 


0 - 


0 


4. 


0 


0 


0 


29. 


0 


0 ' 


0 


5. 


0 


0 


0 


30. 


0 


0 


0 


6. 


0 


* 

0 


0 


31. 


0 


0 


0 


7. 


0 


0 


0 


32. 


0 


0 




8. 


0 


0 


0 


33. 


0 


0 


0 


9. 


0 


0 


0 


34. 


0 


0 


•; 0 


10. 


0 


0 


0 


35. 


0 


0 


D 


11. 


0 


0 


0 


36. 


0 ~ 


0 


o \ 


12. 


0 


0 


0 


37. 


0 


0 


0 


13. 


0 


0 


0 


38. 


0 


0 


0 


H. 


0 


0 


0 


39. 


0 


0 


''0 


15. 


0 


0 


0 


40. 


0 


0 


0 


16. 


0 


0 


0 


41. 


0 


0 


0 


17. 


0 


0 


0 


42. 


0 


0 


0 




0 


0 


0 


43. 


0 


0 


0 


19. 


0 


0 


0 


44. 


0 


0 


0 


20. 


0 


0 


0 


45. 


0 


0 


0 


21. 


0 


0 


0 


46. 


0 


0 


0 


22. 


0 


0 


0 


47. 


0 


0 


0 


23. 


0 


0 


0 


48. 


0 


0 


0 


24. 


0 


0 


0 


49. 


0 


0 


0 


25. 


0 


0 


0 


50. 


0 


0 


0 
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Appendix C ^ 

Observation Test , 
Interview Model, B 
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TLST ON ASSESSING 'HE BELIEVABILITY OF OBSERVATION 

STATEMENTS ^ 

OBSERVATION TEST INTERVIEW MODEL , B 



STEP I . 

- Inform examinee of our purpose: to attempt to develop the 
best test we can of people's ability to think in a certain 
area. 

H 

- Inform examinee of his or her role: to give us information 
about how people think when they take our test so that we 
can change the test where conges need to be made. 

- Inform examinee that we are interested in finding out about 
the test and not about the person taking it, so there is no 
reason to feel any stress or pressure. 

- Inform examinee that we want to find out as best we can what 
he or she is thinking while choosing answers to the questions, 
and that to do this we will be asking some questions as the 
test is written, 

• » 

STEP II 



A. Instructions to the examinee: 



M As you do each question tell me all you can about what 
you are thinking while you are picking your answer. 1 1 

B. Interruptions in the examinee's narrative: 

Only the following are leg itimate* interrupt ions into the 
examinee's thinking and only then when made without 
hesitation: 

- A probe for ambiguous reference of demonstratives or 
third person pronouns by saying: 

"Could you tell me what you mean by . . . ?" 

Example: When talking about two maps the examinee 

says: "On this one here is states clearly 
that ... ", and it is not clear which map 
the student means, it is legitimate to 
probe immediately: "Could you tell me what 
you mean by 'this one here'?" 

- Probe tor obvious reading mistakes by saying: 
"Did you read ?" 

Example: Examinee reads: !f Mr. Wang's car did come 

to a full stop" for "Mr. Wang's car did not 
come to a full stop". It is legitimate to 
probe immediately: "Did you read, 'Mr. 
Wang's car did come to a complete stop'? 
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C. In response to examinee probes: 

i 

Only the following are legitimate types of responses to 
examinees' probes for facts or reasons: 

- If examinee probes for facts, say: "You can only go by 
what is written". 

Example: The examinee says: "For 20, eh, how lon^ 

was this after the accident". Tt is legitimate 
to answer only: "You can go only by what is 
written." 

- if the examinee probes for reason, say: "You can decide 
only according to what is said and what you know." 

Example: The examinee says: "How can you tell if it's 
a shadow or not?". It is legitimate to answer 
only: "You can decide only according to what 
is said and what you know". 



D. Cautions: 

- Do not rush the interview fey beginning to speak immed- 
iately after the examinee ^tops speaking. Wait for a 
few seconds for the examinee to continue, 

- Do not cut off Nominees 1 reasoning by signalling that 
enough has been said, even though many examinees will 
appear by the tone of their voices to seek such signaJi. 

- Do not endorse examinees' fact finding or reason giving. 



Stop at the first level in which the antecedent is fulfilled: 

A. If "neither" chosen, say: "So vou believe neither is more 
believable?" 

B. If the question is an inference vs. observation question, 
then: 

1. it' a criterion is identified as such, if a comparison 
on the basis of that criterion is made, and if a 
general principle on the difference that criterion 
makes is identified, go on to the next question; 
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2. if a criterion is identified as such, and if a com- 
parison on the basis of that criterion is made, then 
probe; "So (state the criterion mentioned) makes the 
difference?", given with emphasis 6n the statement 
of the criterion; 

3. if a criterion is identified as such, but no comparison 
on the basis of , that criterion is made, then probe: 

• "Could you tell me more about the difference (state 
the criterion mentioned) makes to your thinking?"; 

4. if a criterion is not identified as such, probe: 
"Could you explain a little more what makes you believe 
one more than the other?" 

D. If the question 4s not an inference vs. observation question, 
then: » 

1. if a criterion is identified as such, if a comparison 
on the basis of that criterion is made, and if a 
general principle on the difference that criterion 
makes, is identified, go on to the next question; 

2. if a criterion is identified as such, and if a compar- 
ison on the basis of that criterion is made, then 
probe: "So (state the criterion mentioned) makes the 
difference?", given with emphasis on the statement of 
the criterion; 

3. if a criterion is identified as such, but no comparison 
on the basis of that criterion is made, then probe: 
"Could you tell me more about the difference (state 
the criterion mentioned) makes to your thinking?"; 

4. if no criterion is identified as such, then probe: 
"Did (state the criterion) play any part in your 
thinking?" I7~the response is affirmative, then 
probe: "Could you explain the part it played?" If 
the response i's negative, go on to the next question; 

5. if tb^ criterion is identified but rejected as such, 
then probe: "Could you tell me some more about why 
(state the criterion) does not make a difference to 
your thinking?". 

li. Interviewing principles: 

t 

1. if more than one criterion is mentioned including the 
criterion, then probe about the criterion; 

2 . if more than r .e criterion mentioned not including thtT 
criterion, th» n probe for the first criterion; 

5. if there is doubt about categorizing a response, opt 
for the less leading choice, the one which comes first 
in the list. 
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After all the assigned questions are completed, then if there 
is time and "for a maximum of four questions, do the following: 

1. if for an item (after probing) a comparison on the basis 
of a criterion was made but no general principle was 
stated, then probe: "Do you believe that (state \ general 
principle based on the criterion meririonedjVl 

I J if for an item (after probing) the criterion did not play 
a role in the thinking, then proBe": "Could you use (state 
the criterion) to help you decide which is more believable?" 
TFnegative, probe why; if positive, probe how. 
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C.f. Interview Sheets 
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Teat On Assessing the Reliability of Observation Statements, 
/l ' Version B * 

Interview Sheet, Al-15 

* 

TAFE SIDE , SCHOOL < 



GRADE 



QUESTION 


Q-TYPE 


» 

CRITERION 


STUDENT RESPONSE ' 
CRITERION COMPARISON PRINCIPLE 


1 


Other 


Pierre map 
reader 






- 


2 


Other 


small* boy 


• 






3 


Other 


Ms • Vernon 
by-stander 








ti 


Other 


Mr, Dave sure 
Ms, Vernon 
vill agree" 






* 


5 


Other 


Mr. Peters 
upset 








6 


Other 


Ms. Vernon^* 
driver 
education 
instructor 






• 




Other 


Martine hears 
of f i©er 1 s 
leading 
question 


I 


* 




8 


Other ' 


Mr. Dave 
forgot 








9 


Other 


Tvin headlights 
non-salient 


f 






:o 


Other 


4 

Ms. vernon says 
simultaneous 
crash strange 










Other 


Mr. Dave responds 
to leading 

question 






* 




Other ' 


Mr. Dave over- 
hears conversation 








r — — ■ 


Other 


Mr. Dave hitting 
fruit stand more 
salient A 






* 


■ c- - - — — 


Other 


4 » 

Ms . Vernon responds 
at scene of accident 




r 




> c; 


Cther 


Policevoman told 
Martine about 
signalling 

15 


2 




s 

' * 
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Test on Assessing the Reliability of Observation Statements, 
* Version B 

Interview Sheet, A16-28 
TAPE SIDE SCHOOL 



GRADE 



QUESTION Q-TYPE CRITERION 


STUDENT RESPONSE 
CRITERION COMPARISON PRINCIPLE 


16 


Other 


Suzuki made 
measurement 

Y\ n o+" pq 

CU1LL ilU UCO 


I tr 




« 


17 


Other 


Mistake with 
measuring stick 
more likely 








18 


Other 


Microscope 
more precise 






T~ 

i 


• 19 


Other 


O'Reilly uses 
microscope 
regular lv 








20 


Other 


Smith's camera 
older and more 

(III I lLUXl 1>U 

adjust 


• 






21 


Other 


Rousseau took 
pictures W pileup 






■ 


22 


Inference 










23 


Other 


Suzuki only glances 
at picture 










Other 


Rousseau checked 
half an hour ago 








25 


Other 


Suzuki only kneels 
down and peers 

Ui.der car 








26 


Other 


O'Reilly always 
thinks Suzuki is 
wrong 








* 


Other 


Rossi and Suzuki 
found fluid left 










Other 

<< 


O'Reilly reports 
what policeman said 




/ 

s 
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Test on Assessing the Reliability of Observation Statements, 

Version B 

Interview Sheet, B29-3T 
TAPE SIDE SCHOOL 

GRADE 



QUESTION 


Q-TYPE 


CRITERION 


STUDENT RESPONSE 
CRITERIok COMPARISON PRICIPLE 


29 


Inference 










30 


Other 


Cheng says it 
only seems to 
split 


1 






31 


Other • 


Ginette agrees 
with Cheng 








32 


Other 


Mary only gets 
a quick glimpse 








33 


Other 


Scott is upset 








3h 


Other 


Scott often 


0 






35 


Other 


Cheng was 
probably asleep 




s 




36 


Other 


Scott had been 
reading about wild 
animals 








37 


Other 


* 

Scientists more 
believable than 
Scott's friends 


i 

V 

• 
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Test on Assessing the Reliability of Observation Statements, 

Version B 

Interview Sheet, B38-50 

TAPE SIDE SCHOOL 



GRADE 



QUESTION Q-TYPE CRITERION 


STUDENT RESPONSE 
CRITERION COMPARISON PRINCIPLE 


38 


Other 


Cheng emotional 
about pollution 








39 


Other 


Cheng sees stream 
by house nearly 
every day 








hQ 


Inference 






v . 




Ul 


Other 


Cheng very excited 








h2 


Other 


Voices coming from 
all directions 








»»3 


Inference 










U 


Other 


Record mor? precise 
than method allows 








k5 


Other 


Report made day 
after observation 








~6 


Other 


Rocks says tempera- 
ture should be higher 




•i 




1 T 
^ 1 




Five separate readings 
averaged 








U8 


Other 


Ginette sees animal 
twice 








U 9 


Other 


Cheng fond of 
Juan it a 








5C 


Other 


Scott makes statement 
at scene of observa- 
tion 




1 






9 

ERIC 



127 



Appendix D 

Test on Appraising 
Observations, 
Version C 
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TEST ON APPRAISING OBSERVATIONS 

VERSI^JN C • ■ 
by Stephen P. Norrls and Ruth King 
Institute for Educational Research and Development 
Memorial University of Newfoundland 

1983 



This test tells you two stories. Read them very 
carefully. As you read the stories you will be asked to 
answer questions about what people say. You must read ALL 
the Information you are given. EACH piece of information 
may be needed to answer some questions. Each question has 
only QNE accepted answer. To answer a question, do NOT use 
Information given In later questions. You MAY use 
Information given In earlier questions. 
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PART A 



•A Traf f !c Acc Ident 

A traffic accident has Just occurred at an Intersection 
which has a,, stop sign In each direction. Several cars were 
Involved and there were several bystanders. 

A policeman and a policewoman will question people. 
Later several Investigators will collect Information about 
the accident. It Is your Job to Judge the evidence given 
In the statements that follow. 
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I ns truct I ons 



In each question you will be given two under I I ned 
statements . You must choose which under I I ne^ statement. If 
either, vnn have MORE re ason to believe at the time the 
statements are made. 

Remember: Choose between the underlined statements 
only. You may use statements which are not underlined to 
help you choose . 

Here Is an example: 



0. Two friends, Cathy and Helen, are drj.vlng along a 

country road. Suddenly an animal runs In front of the 
car and crosses to the other side of the road. 

Cathy says, "Look! There Is a small brown animal!" 

Helen says, "Cathy, you are wearing dark-coloured sun- 
glasses. That animal was arev." 



To answer this question, first look for some Important 
difference between the people or the situations. In this 
.case, Cathy Is wearing sunglasses. Cathy's sunglasses couM 
have made the animal appear a different colour. From what 
Helen says, It seems that she Is not wearing sunglasses. 
Therefore, Helen would have a better view than Cathy. 
People who have a better view of things tend to be more 
be I I evab I e . 
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Since you ha-ve MORE reason to believe the SECOND 
underlined statement, Helen's, -at the time the statements 
are made, you should mark your answer sheet like this: 



0. 



F I rst 
0 



Second 



Neither 



In the rest of the test questions, mark your answers as 
f o I I ows : 



F I RST: 
SECOND : 
NE I THER 



You have more reason to believe the FIRST 
statement at the time the statements are made 

You have more reason to believe the SECOND 
statement at the time the statements are made 

You have no more reason to believe EITHER 
statement at the time the statements are made 



STOP: 



Wait for the signal to begin question 1 



DO HQl WRITE ON THIS BOOKLET. 



130 



132 



2. 



3. 



A policeman Is questioning Pierre and Martlne. They were 
In their car at the Intersection but were not Involved tn the 
accident. Martlne Is the driver and Pierre, who had been 
trying to figure out which way to go, Is the map reader. 

The policeman asks Martlne how many cars were at the 
Intersection when the accident occurred. She answers, 
"There were three cars . " 

Pierre says, "No, there were f I va cars." 



A small boy, who had been standing next to Mr. Dawe, a 
bystander, says, " There was a motorcycle at tha Inter - 
section . " 

Mr. Dawe says, "No, there was no motorcycle at the Inter 
sect I on . " 



A policewoman has been asking Mr. Wang and Ms. Vernon 
questions. She asks Mr. Wang, who was one of the people 
'nvolved In the accident, whether he had stopped at the 
stop sign. 

Mr. Wang answers, "Yes, I came t o a full stop at the stop 
5 lgn . " 

Ms. Vernon, who had watched the accident happen, tells 
the officer, " Mr. Wang's car did not coma to a full stop.' 
But this didn't cause the accident." 



The policewoman then points to Ms. Rosen's car which was 
one of the cars Involved In the accident. She asks 
whether Ms. Rosen had signalled. 

Mr. Dawe says, " Ms . Rosen signal led . I was Just talking 
to Ms. Vernon and I'm sure she will agree with what I 



Martlne says, " Ms . Rosen d I d not s I gna I . I'm sure I'm 
right." 
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The policeman talks to Mr. and Mrs. Peters, who were also 
involved In the accident. It Is easy to see That Mr. 
Peters, who was the driver, Is very upset by the 
accident. The policeman asks him to estimate his speed 
Just before the accident. 

Mr. Peters says, "I was g oing about 15 kilometers an hour . 

A little later when he Is feeling better he says, " I was 
going about 30 kilometer s an hm.r ." 



The policeman asks whether or not the Peters' car had 
stopped at the stop sign. Ms. Vernon, who Is a driver 
education Instructor, says, "I am very experienced In 
these matters. The Pet ers' car did not stop ." 

Martlne, who overneard this conversation, goes up to the 
officer and says, "The Pat ers' car did stop at the stop 
sign." 



The officer turns to question Martlne and Pierre and Mr. 
Dawe. The officer asks them to estimate the speed of 
Mr. Wang's car when It hit the others. 

Mr. Dawe says, "It was going ah out 40 or 45 kilometers 
an hour." 

The officer says, "It was going faster than that, wasn't 
It?" Martlne says, " Oh yes r It was going about 60 or 65 
k 1 1 ometers an hour . " 



Martlne adds, "Mr. Wang went r ight through the stop 
sign." 

The police officer turns to Mr. Dawe and says that at 
the scene of the accident Mr. Dawe couldn't remember 
whether Mr. Wang had stopped at the stop sign or not. 
Mr. Dawe says, "I remember now, Mr. Wang did stop at 
the sto p sign ." 
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9. Ms. Vernon then says, | also remember that a fancy b I ue 
sports car went through + h« stnn sign. " 

i 

Martlne says, "A car with twin h eadlights want right 
through the stop sign." 



10. Mr. Dawe says, "Three cars collided at the same time. 
There was one crash," 

Ms. Vernon says, "There was more than one crash. It 
would be very strange for the three to collide at 
exactly the same time." 



11. The police officers ask the people Involved In the 
accident and the witnesses to come to the police 
station to make official s tat erne n ts . At the station, 
the policeman questions Mr. Dawe. 

Mr. Dawe says, " Just before th e accident occurred Mr. 
Wang'£ f Ms, Rosen's, and Mr. Peters 1 cars approached 
the 1 ntersect Ion , " 

The police officer asks, "Surely you saw other cars, 
Mr. Dawe?" "Oh yes," says Mr. Dawe, "t here was another 
car . 



12. In the background there has been a conversation between 
the other officer and some of the other witnesses. Some 
are discussing whether one car went through a stop sign. 

Mr. Dawe heard this and continued his testimony, "Mr. 
Wang and Ms. Rosen crashed Into each other. I saw It 
happen." 

"Also, I remember that a car went straight through a 
stop s 1 g n , too." 



13. Nearby, the policewoman Is questioning Martlne. 

Martlne says, " A short time before the accident 
everyone was driving normally." 

She continues, "Then there was a loud squeal of tires. 
Mr. Peters 1 car turned quic kly toward the fruit stand." 
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The policewoman asks Mr. Dawe to tefl In which direction 
Mr. Peters was travelling before the accident. Mr. Dawe 
says, "He was going tn M arri S Fifth Street ." 

The policewoman looks at her notes which were made at 
the scene of the accident. At that time Ms. Vernon had 
pointed and said that Mr. Peters was going away from 
Fifth Street haf ore the accident . 



The policewoman remarks that many people do not use 
their direction signals at Intersections. She says that 
this causes- many 'accidents. She asks Martlne to 
continue to tell what she remembers. 

Martlne says, "Ms. Rn^an rams t o a mmp I ate stop ." 

She then adds, "But she did not use her signal ." 



Meanwhile, at the scene of the accident several 
Inspectors have been collecting Information about the 
accident. They are examining the wrecks and the marks 
on the road. Two tenants were collecting 'nformatlon 
separately. They are now finished and are comparing 
notes . 

Inspector Suzuki says, "Our notes say that Ms. Rosen's 
car skidded 427 centimete rs before hitting the other 
£a_T_&. I made the neasurement and also made the notes." 

Inspector Rousseau s"ays, "According to our notes Ms . 
Rosen's car skidded 457 rentlmeters before hitting the 
Qther cars. Inspector O'Reilly measured the skid by 
herself and Inspector Smith copied down what she said." 



Inspector Rousseau fays, "We also measured the length 
of Mr.. Wang's skid. We used a measuring tape that was 
1000 centimeters long. Inspector O'Reilly held one 
end at the beginning of the skid and I took the reading 
at the other end. I wrote down the measurement. Mr . 
Wang's car skidded 32Q cen timeters ." 

Inspector Rossi says, "I also measured Mr. Wang's skid. 
I used a 30 centimeter measuring stick. I started by 
placing one end at the beginning of the skid and by 
putting a mark at the other end. I then placed the 
beginning of the stick at that mark, and so on until I 
reached the end of the skid. I wrote down my measure- 
ment. Mr. Wang's car s kidded 350 centimet ers." 
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18. Inspector Rousseau says, "I found some brown paint on the 
left front fender of Mr. Wang's car. I looked a»t It 
with a magnifying glass. It Is the same colour as the 
paint on Ms. Rosen's car." 

I o-specto'r Ross I says, "I also studied that paint on the 
left fender of Mr. Wang's" car. I looked at It under the 
microscope. It Is not the same col our as the paint on 
Ms. -Rosen's car." * 



19. Inspector Smith, who does not use a microscope often, 
says, "I'd like to check that myself." He looks at the 
paint sample under the microscope and says, "There are 
no gold-coloured spots In this sample." 

Inspector O'Reilly, who uses a microscope often, looks at 
the sample. " There are go I d-co loure d spots In the 
samp I e." she says. 



o 

20. Inspector Rousseau and Inspector Smith have been using 
cameras which develop pictures Instantly to take 
pfctures of the accident. Inspe-ctor Smith's camera Is 
an older model and Is more difficult to adjust. They 
compare pictures of the skid marks of. Ms. Rosen's and 
Mr. Wane's cars. They are trying to^ find out who 
stopped faster. 

Inspector Smith points to his pictures and says, "Mr.. 
Wang's skid marks a re darker than'Ms. Rosen's." 

Inspector Rousseau looks at his pictures and says, "No, 
Mr. Wang's skid marks are no dar ker than Ms. Rosen's 
skid marks." 



21. Both Inspector O'Reilly and Inspector Rousseau have 
taken pictures of the cars Involved In the accident. 
I nspector 0 ' Re I I I y says, "My pictures show that Ha. 
Rosen's and Mr. Wang's cars were damaged the same 
amount . I took several pictures of each car by Itself 
after they were rolled apart." 

Inspector Rousseau says, "My pictures show that Ha. 
Rosen's car was damaged more than Mr. Wang's car.- I 
took several pictures of the pile-up before the cars 
were rol led apart." 
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22. Inspector O'ReTNy says, " Mr. and Mrs. Pftfe rs' car Is 
on I y gj Iflht I y damaged . 

She continues, "The accident pr obably wasn't Mr. Peters' 

fault." 



23. Inspector Rossi and Inspector Suzuki examine the' 

pictures taken by O'Reilly and Rousseau.' Inspector 
Suzuki glances at a picture and says, " There Is a 



Dart hanoJ^Q down under M r. Wang's car 



I nspector 
and says, 
a shadow . " 



Rossi studies the picture for severa 
"That's not part Mr. Wang'* r^r . 



secon d s 
That 's 



•3 



24. They then turn to examine the 
Rossi points and says, "Look, 
front brakes of Ms. Rosen's rar 



wrecked cars. Inspector 
the brake line to the 



Is broken 



it 



""Snspector Rousseau overhears 
strange. I discovered about 
brakellne was not broken ." 



this and says, "That's 
an hour a,go that that 



25. Inspector Smith s-lldes under Ms. Rosen's car to examine 
the brakellne. V The handbrake cable Is bro ker" he says. 

Inspector Suzuki kneels down and peers under Ms. Rosen's 
car. "No," she says, " the handbrake ga m ble Is not 

broken ." 



26. Inspector Suzuki examines the brakellne of Ms. Rosen's 
car. She says, "This rubber hose In the brakellne Is 
worn through. It must have happened gradually." 

Inspector O'Reilly, who thinks that Inspector Suzuki Is 
always wrong, also exam Lies the brakellne. "No," she 
says, "the rubber hose Is cut. It must have snapped 
suddenly." 
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27. Inspector Rossi checks the brake fluid container of 
Ms. Rosen's car. He tells the other Inspectors that 
there Is a small amount of fluid left. 

Inspector Smith checks 'he fluid container as well and 
says, "Thprp Is no fluid left there. " 

Inspector Suzuki checks as well and says, " Thara Is a 
LLtlifl Lflll a+ the hottnm." 



28. Inspector O'Reilly says, "One of the police officers 
checked the brakes. He told me that he pressed the 
brakes and they worked . Ms. Rosen had at least partial 
braking power at the time of the accident." 

Inspector Rousseau sa> "I Just checked the brakes 
rruyself. I pressed the brakes and the pedal went 
stra . g ht to the f I oor . Ms. Rosen had no braking power 
at the time of the accident." 



The Investigators were eventually able to agree on all 
aspects of the Investigation. They turned their report over 
to the Insurance company. 



STOP HERE. THIS IS THE END OF PART A. 

DO NOT GO ON UNTIL YOU ARE TOLD. IF YOU HAVE TIME, CHECK 
YOUR ANSWERS TO THIS PART OF THE TEST. 

IN PART B A NEW STORY BEGINS. 

THE INSTRUCTIONS ARE THE SAME AS FOR PART A. 
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PART B 



Exp I or I ng a River 



Imagine that you are a member of a group which Is 
exploring one of your country's rivers. There are several 
people In the group who have no advanced scientific 
knowledge. There are also two scientists In the group. 
Professor P l ant , a biologist. Is an expert In the study of 
living things: plants. Insects, animals, and so on. 
Professor Rocks, a geologist, Is an expert In the study of 
the Earth: the structure of rocks and minerals and the 
shape of Its surface. Their names should help you to 
remember the subjects I r, which each Is an expert. 

You have Just arrtvec at a camping place by the side of 
the river. It Is early morning and Just becoming bright. 
People In your group bsgln i-o explore the area around the 
camp. 

In what follows the rest of your f r ; p w ||| be described. 
You will be glv*n statements made by people In your group 
during your trip. As In Part A, you will be asked to Judge 
how believable these statenents are. 
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You and some members of your group are getting a fire 
ready to cook breakfast. Others are looking at some 
mountains which are several miles away. Juanlta says, 
" Those m ountains have several white streaks going from 1 
the top to the bottom." 

Professor Rocks says to her, "Those streaks are small 
streams , I would say." 



Cheng says, " The white streak farthest to the right 
seems to divide Into two parts halfway down the 
rnqu n ta Ins . " 

Scott says, " It does divide Into two parts at that 
- >o 1 n t . Some of j/t goes In one d I rect I on, the rest 
goes I n another . " 



Professor Plant says, " It does not seem to divide Into 
two parts. ; ' 

Glnette says, "I think Cheng Is right. I t rioes seem to 
d ivide 1 nto tw o parts . " 



Meanwhile, Mary, Juanlta, Scott, and Cheng are walking 
through the campsite. Scott points to his right and 
says, "Look, there ar e two Swa I I ows ." 

Mary, who had been looking to Scott's left, turns 
quickly In the direction Scott pointed. She gets a 
qul:k glimpse of the birds and says, "Those birds are 
not Swallows. They are Ch l ckadees ." 



Juanlta says, "I saw the birds and Mary was right, 
Scott. The birds were Chickadees." 

Scott becomes upset at what Juanlta says. He shouts, 
"1 know what I saw. The b lrds were Swal I ows ." 
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34. Scott quickly calms down. Juanlta Jokes and reminds 
hTl m that he often confuses birds. For example, 
yesterday he pointed to some Sparrows and said they 
were Junrros. Everyone had agreed he was wrong. Scott 
laughs and agrees that this was so. 

Juanlta then says, "The birds to which Scott pointed 
this time were Chickadees , I had a good look at them. ft 

Scott responds, " Mm sure this time they were Swallows , 
I got a good look at them, too. 11 



35. Scott and Cheng leaye Mary and Juanlta. They Join 

Professor Rocks and lie down near the edge of a large 
grove of trees. Cheng says he fs sleepy and closes. his 
eyes. Scott becomes fascinated by a book called Wild 
Animals of North/ Amer 1 ca . It contains things he never 
heard before. Professor Rocks Is watching the sky. 
About one half hour passes. Cheng has not made a sound 
the whole time. His eyei are still closed. Scott Is 
I ook 1 ng at his book • 

Suddenly Scott says to Cheng, " I hear a s ound In those 

trees behlruLjis • " 

Cheng says, " There was no sound In those trees . Stop 
talking." 



36. "There was a'sound," Scott Insists. " ) t was a b I ackhear. " 

"I heard a sound, too," says Professor Rocks. "But 11 
was not the sound of a blackbear . " 



37, Professor Rocks continues, "According to Professor 

Plant, a team of sclentjsts studied the wildlife of this 
area last summer . In their report they IPsted all the 
animals they sighted. They reported that they saw no 
b 1 ackbear s . " 

"Several of my friends visited this area last summer," 
replies Scott. "One told me In a letter that they saw 
some b I ackbears . " 
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38. Scott and Professor Rocks agree to check with Scott's 
friend about seeing bears. They hear no more sounds. 
You finish your breakfast and start your hike down- 
stream. In a little while Scott points to a small stream 
flowing Into the main one. This stream Is not pleasant 
looking. Its water Is coloured orange. 

Cheng Is a member of a group which Is fighting to stop 
pollution of rivers. He sees the coloured water and 
become** very angry. He exclaims, "Some people are very 
careless! There Is not a plant living In that water." • 

Mary Is a member of the same an t l-po I I ut Ion group as 
Cheng. She looks at the water too and says, " Th are 
are some plants living In the water." 



39. Cheng looks at the water again and says, " There are no 
Insects In the water." 

He says, "There Is a stream close by my house which I 
see nearly every day. It Is dirty like this one. There 
are no Insects In that stream, either." 

S 

40. You continue down the river. Before long Scott yells, 
• " I sme I I smoke 1 I wrnder If It's a forest fire?" 

Everyone stops and sniffs the air. Cheng says, "Yes, 
It Is a forest fire ." 



41. You and your group decide to discover the source of the 
smell. You find a path leading through the'forest In 
the direction from which It Is coming. Cheng rushes 
ahead of the others. 

Cheng screams, " I see smoke up ahead ! The forest Is on 
fire!" 

GInette catches up with him, looks and say, "No, Cheng, 
that's not smoke . It Is fog." 
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As you walk along the path, you reach an area where you 
hear echoes of your voices from all directions. 

Mary says, "I hear a knocking sn.mri straight ahaaH ." 
Professor Rocks says, "I see someone str aight ahead ." . 



Your group rounds the next bend and Juanlta says, "Look, 
there are some campers. They have a eampf Ire burning. " 

Glnette says, "I would say the smok e we ware sme I I I ng 
came fro m there ." 

You continue down the path until you reach a log cabin. 
The owner Is outside working on the woodpile. He greets 
your group and Invites you all to stay for lunch. You 
accept . 



While you are resting after lunch. Professor Rocks 
notices that Juanlta, Glnette, and Scott are Interested 
In the rocks of the area. She reaches Into her knap- 
sack and takes out two old notebooks. She says that 
the notebooks contain the records of two different 
scientific teams. Both teams were studying the rocks 
of the area when they made the records. Juanlta, 
Glnette and Scott start to look through the records 
and to read some of the reports. One Is a report of 
Lookout Mounta I n . 

Juanlta says, "At the beginning of the record I am 
reading the writer says that the lake below Lookout 
Mounta fn .-l&.JJ 54. centim eters dee p. He says his team 
found the depth by lowering a string with a lead weight 
tied to It Into the water until It reached the bottom. 
Then they measured the wet part of the string." 

Glnette says, "This record reports that the lake ha|pw 

Lookout Mountain is aJa.QUji.J-lAQ>j£fi.gj--lflis.t.er-5^flflP They 

describe their measuring method. It Is the same as the 
one described In the record from which Juanlta Just 
read." 



142 



144 



45. Juanlta says, "The record I am now reading says that 
the following report was made the day after the 
observation was made. According to the record there Is 
a hot spring behind Lookout Mountain. It reports that 
the hot spring shoots out water every thi rteen minutes . " 

Glnette says, "This record says that each report was 
made within five minutes after the observation was made. 
It also talks about the hot spring behind Lookout 
Mountain. However, It says that It shoot s out water 
every nine minutes." 



46. Glnette says, "This record 
temperature of the .hot. .SP_C_ 



says that the 
ng is 11 ° C." 



water 



Professor Rocks remarks, "That Is strange. A hot 
spring should have a much higher water temperature 
than that." 
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Ju an l/fa says, 
Is 60° C " 



"This record says that the temp erature 



47. Scott says, "This record reports the temperature of the 
water In a small stream near the hot spring. The 
report says that five separate readings were taken and 
the average of the five readings recorded. It reports 
that the temperature of the warer Is 25° C." 

"This record also reports the temperature of a small 
stream near the hot spring. It does not say whether 
more than one reading was taken. It reports that the 
temperat ure of the water Is 20°C." Professor Rocks 
says. 



48. At this point everyone Is well rested. You 
follow a path which leads around the lake, 
an animal crosses your path. 



begin to 
Suddenly 



Juanlta says, "Look, there's a small red squirrel." 



Glnette says, " 
little and says , 
red squ I rre I . " 



I see It, 



see 



too!" She runs after It a 

t agalnl That an 1 ma 1 Is not a 
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49. Cheng, who Is very fond of Juanlta and often does 

things to try to make her think highly of him, says, 
"There's another animal over there. It Is a red 
squ I rre 1 like you were hoping to see, Juanlta." 

"No," says Scott, "That ani mal Is a shrew . " 



50. You continue along the path. Cheng recalls, "Remember, 
Scott, we pitched our tent a short way from here last 
summer. We could see that mountain to the right of our 
tents." 

When you reach the camping spot, Scott says, "No, Cheng. 
Our tent was here. We could see the mountains to our 
left , there." 



In another hour you come to a road. You have reached the end 
of your trip. A person Is waiting to drive your group home 
as you j had planned. 



THIS IS THE END OF THE TEST. 

CHECK YOUR ANSWERS TO THIS PART IF YOU HAVE TIME. 
DO NOT GO BACK TO PART A. 
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Appendix E 

Instruction Sheet to 
Cooperating Teachers 
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GENERAL DIRECTIONS TO TEST ADMINISTRATORS 

1. The, Norris-King test should be written first. 

2. Each class should write the Norris-King \$st and ONE of either the Cornell Test 
or the Watson-Glaser Test, for a total of N^wo tests. 

3. All three tests are power (untimed) tests, \here are no specific time limits, 
but most students finish in 45 to 50 minutes A 

4. Student responses are confidential. However, we\do need to be able to identify 
1n some way the two tests written by the same student. Please ensure that 
either the students' names or code numbers appear \m the answer sheets. 

5. All three tests have separate answer sheets. Please\ask students not to write 
on the question booklets. \ 

6. None of these answer sheets will be computer scored. . Therefore, you can ignore 
any directions concerning a particular type of pencil lead. 

NORRIS-KING TEST 

■ ♦ 

1. Please reuci the directions on the cover page and the Directions page with the 
students. Emphasize the bottom four lines of the Directions page. 

2. When students reach the end of Part A (question #28), they may continue on to 
Part B without waiting for further instruction. 

WATSON-GLASER TEST 

1. Students can be told to ignore the section of the answer sheet which asks for 
their name in computer legible form (the letter boxes). However, they must 
write in either their name or code,- as you have chosen. 

2. Please read, with the students, the directions on the cover page and page 2. 
Emphasize to them that the four remaining subtests have slightly different 
directions, which they should read as they work through the booklet. You 
need read with them only the directions to TeSjt. 1. 

CORNELL TEST 

1. This test is organized as Part I, Section A (items 1 to 25), Part I, Section B 
(items 26 to 50), Part II, Section A (items 51 to 65) and Part II, Section 3 
(items 66 to 76). 

2. Please read with the students the cover page and page 1, the directions for 
Part I, Section A. Tell them that each of the remaining sections has a 
different set of directions, which they should read themselves as they come to 
each. 

3. Emphasize that, for both sections of Part I, (items 1 to 50) they should nut 
return to a question cnc? they have passed it. However, in Part II (ite 3 
51 to 76), thdy r.ay return to iteT3 if they wish. 
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Appendix F 

Key to Correct Answers and 
Principles Tested per Item ' 

on the 

Test on Appraising Observations 



147 



150 



It pm 


Prinpinlp 

i 1 iil wpiv 


Keved 


Item 


Principle 


Keyed 


m hpr 


Tested 

x CO vCU 


ResDonse 


Number 


Tested 


Response 


1 
1 


n 2 

Li > M 


FIRST 

1 AA <J A 


26 


n.io 


FIRST 


0 


n 12 

LI. 1 *f 


SECOND 


27 


IV.2 


SECOND 




n 3 


SECOND 


28 


IV.5 


SECOND 






FIRST 

X X 


29 


I 


FIRST 




n i 


SECOND 


30 


IV. 1 


FIRST 


A 
\j 


n 4 


FIRST 


31 


rv.2 


SECOND 


m 

I 


rv ii 

& T » X A 


FIRST 


32 


m.2 


FIRST 


o 


rv 12 

IT » 1 w 


FIRST 


33 


m 


FIRST 


Q 


rv 13 

I T » X V 


FIRST 


34 


n.7 


FIRST 


10 


n 5 

11 > KJ 


SECOND 


35 


n.2 


FIRST 


1 1 


IV 11 


FIRST 

X JUL V ' X 


36 


n.io 


SECOND 


1 0 


ii > 1 1 


FIRST 


37 


rv.Hc 


FIRST 


io 


rv 13 

IT t A KJ 


SECOND 


38 


rv r .9 


SECOND 


1 4 


rv 8 


SECOND 


39 


m.3 


SECOND 


] 

i *j 


ii > 1 1 


FIRST 


40 


i 


FIRST 


18 


rV 14b 

IT t X s U 


FIRST 


41 


rv.9 


SECOND 


1 7 


n s 


FIRST 

X XX l>kj X 


42 


m.i 


SECOND 


IS 


ITT 4a 

lll.TCl 


SECOND 


43 


i 


FIRST 


1 Q 


n q 

Li, <7 


SECOND 


44 


rv.3 


SECOND 


to* v/ 


DI.4e 


SECOND 


45 


rv.4 


SECOND 


•21 


m.4e 


FIRST 


46 


rv.7 


SECOND 


•>o 


I 


FIRST 


47 


11.8 


FIRST 


23 


in. 2 


SECOND 


48 


in.3 


SECOND 


•24 


IV. 4 


FIRST 


49 


n.3 


SECOND 


'25 


IU.'J 


FIRST 


50 


rv.8 


SECOND 



148 



